Approach Paper Engaging Citizens for Better Development Results An IEG Evaluation of World Bank Group Citizen Engagement April 4, 2017 Background and Context 1. This proposed evaluation aims to inform the Board and Management of the World Bank Group on the effectiveness of World Bank Group citizen engagement (CE) activities to support development processes and outcomes. The evaluation is intended to provide a timely contribution to management’s review and update of corporate CE goals, targets, and approaches upon completion of the 2018 corporate CE commitment. This is the first IEG evaluation that systematically reviews the World Bank Group CE efforts (including IFC and MIGA). 2. CE is the umbrella term adopted by the World Bank Group to denote a multitude of different interactions with a variety of stakeholders —such as government, private sector, and development institutions, including the World Bank Group—at different points in the project or program cycle that give citizens a stake in decision making with the objective of improving development outcomes. 1 It is a two-way relationship that implies the existence of a tangible response to citizens’ feedback. 2 3. Under different names, CE has a long history. The first United Nations Conference on Environment and Development in 1992 recognized that achieving sustainable development would require the active participation of all sectors and all types of people. Many donor and government agencies have expanded their efforts to engage with and seek the participation of external stakeholders often labeled as “citizens,” “beneficiaries,” or “communities.” 4. Stakeholder engagement, as expressed in consultations, social accountability, community-driven development, participatory approaches, and empowerment, has long been an integral part of the World Bank Group’s operational activities, documented as far back as the 1970s. Engagement has included stakeholders at multiple levels, ranging from consultations with client country governments, nongovernmental organizations (NGOs), and civil society organizations (CSOs), and the private sector, to participation of direct and indirect project beneficiaries. The World Bank Group Strategy (2013) describes citizen voice as “a key accelerator for achieving results.” 1 IFC more commonly refers to “Stakeholder Engagement” instead of “Citizen Engagement”. 2 This two-way relationship distinguishes CE from the concepts of “transparency” and “information sharing”. While informing citizens is a necessary aspect of CE, it is insufficient, as CE also requires a response to citizens’ feedback. 1 5. This stream of activities has intensified over the years and culminated in 2014 in the development of a Strategic Framework for Mainstreaming Citizen Engagement in World Bank Group Operations (World Bank 2014), which aims to provide a more coherent approach to incorporating CE across World Bank Group operations. The Strategic Framework will serve as the primary reference point for this evaluation. 6. The objective of the Strategic Framework is to “mainstream citizen engagement in World Bank Group-supported policies, programs, projects, and knowledge and advisory services where such engagement can improve development results and, within the scope of these operations, to contribute to sustainable processes for citizen engagement with governments and the private sector.” The Framework builds on lessons from World Bank Group-financed operations across regions and sectors, and underscores the importance of country context, government ownership, and clear objectives for citizen engagement. It proposes five underlying principles for mainstreaming citizen engagement in Bank Group-supported operations: focusing on results; engaging throughout the operational cycle; seeking to strengthen country systems; applying context specificity; and using a gradual rollout approach. 7. A review of the history of CE in the World Bank Group reveals that the Strategic Framework builds on and brings together several streams of work that have been evolving over decades from an initial focus on reputational risk management to operational engagement to stakeholder influence on policies and strategies at the macro level (Box 1). These streams are: stakeholder consultation in policies and operational instruments (1980 onward); collaboration with beneficiaries to boost local ownership and responsiveness to local needs (1993 onward); and consultation with stakeholders, including beneficiaries, for information sharing and increased voice at the macro level in formulation of policies, strategies, and policy lending (1996 onward). CE is part of a broad move toward greater openness and transparency, as witnessed also by World Bank initiatives around governance and anticorruption, open data, and access to information enacted in the 2007-12 period. 8. In 2013, the World Bank Group committed to integrate beneficiary feedback into 100 percent of projects where beneficiaries can be clearly identified by FY18. 3 The World Bank Corporate Scorecard, the IDA Results Measurement System, and the President’s Delivery Unit track the progress of this commitment. The indicator used to track progress is based on World Bank investment projects and is narrower than the wording of the corporate commitment. 9. The International Finance Corporation (IFC) and the Multilateral Investment Guarantee Agency (MIGA) have their own systems for approaching stakeholder and community engagement. They require their clients to adhere to a set of eight Performance Standards, representing the bulk of their CE activities. The Performance Standards constitute an integrated sustainability and risk management framework. IFC adopted the Performance Standards in 2006 and revised them in 2012. MIGA followed a year later (respectively, 2007 and 2013). The 3 This commitment applies to all Investment Project Financing (IPF) operations financed with IBRD loans or IDA credits. 2 Performance Standards impose requirements for meaningful stakeholder engagement on all IFC and MIGA projects, with variation depending on project type and risk factors. Clients implement the Performance Standards which, among several other things, call for: information disclosure and engagement of local communities on matters that directly affect them; active management by clients of social and environmental risks; and grievance redress mechanisms for affected communities, workers, and any people affected by displacement. IFC and MIGA supervise clients’ implementation of the Performance Standards. The Independent Evaluation Group (IEG) reviews the environmental and social performance of projects using a framework linked to the Performance Standards as part of its validation of IFC and MIGA self-evaluation, a review that includes field visits to select projects. IEG undertakes similar reviews for all cancelled MIGA guarantee projects as well as some active MIGA guarantee projects. Box 1. Citizen Engagement Has a Long History at the World Bank Group Consultation in the World Bank Group policies and operational instruments . Starting in 1980, stakeholder consultation in policies, approaches, and instruments was initially driven by a concern for reputational risk management as the World Bank formulated policies to address the criticism of various forms of adverse impacts generated by World Bank- supported operations. This reactive approach gradually evolved into more proactive engagement with citizens through participatory assessments and deliberate strategies aimed at reflecting stakeholder concerns in project. Collaboration in operational work. Collaboration with beneficiaries in operational work was formalized in 1993 through new language on consultation in operational policies for forestry and water resource management. It was aimed at ascertaining demand and enhancing the voice of user groups in sector operations. Over time, this effort gave rise to partnerships with civil society organizations as service providers. Also, a range of multisector operations aimed at providing goods and services to beneficiary communities through demand-driven operations were brought under the loose umbrella of community-driven development operations. Consultation at the macro level. Consultation with stakeholders in policy and strategy formulation became more systematic after 1996. All major policy reforms of safeguard policies and policies governing new lending instruments, such as development policy lending and program-for-results, involved extensive consultations with borrowers, development partners, and civil society organizations. Stakeholder consultation also became more systematic and routine in the formulation of country assistance and sector strategies. Consultation and participation was also integrated within development policy lending. 10. In addition to overseeing client implementation during the portfolio relationship, IFC does have project-related CE in a limited number of projects where IFC evaluates whether 3 projects have “broad community support” 4 and may follow up directly on client CE in some instances. IFC also engages with stakeholders in policy development activities and Spring and Annual Meetings. 11. The World Bank Group established the Global Partnership for Social Accountability (GPSA) in 2012. Through its grant-making and knowledge and learning activities, the GPSA seeks to expand opportunity for civil society and government actors to collaborate to tackle governance challenges and improve development outcomes, especially in service delivery. Situated in the Governance Global Practice, the GPSA is governed by a Steering Committee comprised of CSOs, government, and donor representatives, and is chaired by the Vice President of the Equity, Finance, and Institutions Practice Group. The program provides grants to CSOs for social accountability work and operates a knowledge and learning platform on the impact of social accountability interventions. The program has allocated about $21 million from World Bank Group funds to 32 projects mostly in health, education, public financial management, and local governance in 25 countries. Grants are embedded in the sectors and areas where the World Bank has an active engagement. Grants are awarded to CSOs only in countries that formally opt- in to GPSA. In addition, trust funds, such as the Japan Social Development Fund, also contribute to the social accountability and CE agenda, as do multilateral initiatives such as the Open Government Partnership. ENGAGING CITIZENS: WHY AND HOW? 12. The World Bank Group rationale for involving stakeholders and citizens rests on the need to improve development outcomes: “Supporting client engagement with citizens where such engagement can improve development outcomes is a key component of the World Bank Group’s strengthened focus on results” (World Bank 2014, 5). This “instrumental” rationale for CE will also inform this evaluation, while recognizing the intrinsic value that the World Bank Group assigns to CE. CE can be used to mitigate unintended adverse outcomes, for example, through safeguards (World Bank) and Performance Standards (IFC and MIGA). It can also be used to better achieve intended positive outcomes, for example, by lending voice to service users, helping tailor services to their needs, and providing avenues to hold service providers accountable. The World Development Report 2004 (World Bank 2003) famously coined the term “short route of accountability” to describe client power in relation to service providers and “long route of accountability” for citizen influence on the state (“voice”), see Figure 1. 13. There are multiple views and approaches to CE and different schools of thought, and the “instrumental” view adopted by the World Bank Group is contested. Where the World Bank Group frames CE as a means to an end, others see participation as a value in itself, reflecting the right and ability of citizens to contribute to the governance process. Sen (2005) argues that the 4 Broad community support is required in cases where the business activity to be financed is likely to generate potential significant adverse impacts on communities or is likely to generate potential adverse impacts on indigenous peoples. In those cases, IFC clients are required to engage in a process of Informed Consultation and Participation (ICP). IFC determines whether ICP by the affected communities is required and whether the business activity will need to receive broad community support. 4 opportunity for citizens to participate in political discussions and influence public choice is central to participatory governance and a critical component of democracy. The approach proposed by the United Nations (2008) recognizes that the “rights” component and the “development management” component are two separate but complementary aspects of public governance. It also highlights that the two perspectives are interdependent: citizens may be involved in defining the goals of development. Figure 1. The Long and Short Routes of Accountability 14. The entry points utilized by the World Bank Group in its analytical and operational work indicate many opportunities to engage with stakeholders. Table 1 shows that entry points exist at every level (corporate, country, project) and type of instrument: in project design, project implementation, and project evaluation; policy design and policy implementation; analytical products and services; accountability mechanisms; and country strategy formulation. Each entry point involves one or more CE activities. The Strategic Framework proposes a classification of CE activities into seven types: (i) consultation; (ii) grievance redress mechanisms (GRM); (iii) collecting, recording, and reporting on inputs from citizens; (iv) collaboration in decision making; (v) citizen-led monitoring and evaluation or oversight; (vi) empowering citizens with resources and authority over their use; and (vii) building citizen capacity for engagement. Each activity in turn may involve a number of specific mechanisms. For example, consultations (category i) may be carried out through public hearings, focus group discussions, or advisory committees. Empowering citizens (category vi) may be achieved through participatory planning, 5 community management, community contracting, or participatory monitoring (World Bank 2014, 24).5 Table 1. Main entry Points for Citizen Engagement Level and Citizen Engagement Entry Point Instrument Dialogue with CSOs Partnership programs, including the GPSA and the Open Government Partnership Corporate Regional strategies Monitoring of corporate indicator Consultations for the preparation of country diagnostics and partnership frameworks (SCD/CPF) Country Identification of CE for the inclusion in country operations Use of CE indicators in CPF results frameworks FINANCING INSTRUMENTS: Projects triggering OPs (mostly safeguards) requiring consultations and GRMs Projects with one or more CE activities (such as consultations; GRM; collecting, recording, and reporting on inputs from citizens; collaboration in decision-making; citizen-led monitoring and evaluation or oversight; empowering citizens with resources and authority over their use; building citizen capacity for engagement), particularly in the IPF and PforR following areas: CDD projects Service delivery projects Governance and public financial management projects Natural resource management projects Government engagement with citizens in the design of reform programs Reporting by the Bank on country’s arrangements for consultations with and participation of key stakeholders in the DPF operation and on outcomes of the participatory process. Poverty and Social Impact Analysis Prior actions and benchmarking related to CE IFC and MIGA Implementation of the Performance Standards Advisory CE in design, elaboration, and evaluation of knowledge products Services and Technical Assistance Analytics Analytic work on specific aspects of CE Source: (World Bank 2014, 22). Note: CDD = community-driven development; CE = citizen engagement; CPF = country partnership framework; CSO = civil society organization; DPF = Development Policy Financing; GPSA = Global Partnership for Social Accountability; GRM = grievance redress mechanism; IFC = International Finance Corporation; IPF = Investment Project Financing; MIGA = Multilateral Investment Guarantee Agency; OPs = Operational Policies; PforR = Program for Results; SCD = systematic country diagnostic. 15. Depending on the type of activities and entry points utilized, the ‘citizens’ may be defined as the ‘beneficiaries’ of World Bank projects (population targeted by the project, whose feedback is required by the corporate indicator), as ‘stakeholder’ communities from the perspective of IFC and MIGA (mostly reached through the client), or even all the citizens of a 5 The Strategic Framework presents a mapping between CE activities and mechanisms described in Table 3.2 of the Strategic Framework (World Bank 2014, 24). This evaluation will adopt terminology and classifications identical to the Strategic Framework to the extent feasible. 6 specific country, when the focus is on country-level activities to strengthen government accountability vis-à-vis its citizens. EVIDENCE ON THE EFFECTIVENESS OF CITIZEN ENGAGEMENT 16. Reviews of the literature on CE point to important variations in the effectiveness of CE depending on context, nature of intervention, and type of outcome (Gaventa and Barrett 2012; Mansuri and Rao 2013; World Bank 2014). CE in areas such as CDD, service delivery, and natural resource management has been extensively studied and a number of comprehensive literature reviews exist. Much simplified, the gist of these reviews is that CE can lead to improved outcomes in some circumstances, but that no effects or even adverse effects are also possible. Much depends on what type of development outcome one looks at, the vehicle for participation, a variety of contextual factors, and the quality of implementation. These issues are briefly outlined here: x Many types of outcomes have been studied. The strongest evidence links CE to improved delivery of public services such as water, health, and education. CE can contribute to increase access to and quality of services, and make them more responsive to user needs. There is also evidence of positive outcomes in areas such as empowerment, social inclusion, and cohesion; local public goods such as public safety; and processes for citizen participation in public financial management and natural resource management. x Results are not always positive. Studies have found no results or even negative outcomes in the form of state failure to respond to citizens’ claims, instances of participatory processes that were manipulative or unrepresentative, and violent oppression of citizen demands. x The vehicle for participation matters. Organic participation through social movements, community associations, and other civic groups emerges spontaneously. These groups often have charismatic leaders and exploit political opportunities to effectively channel citizen voice. In contrast, induced participation refers to initiatives endorsed by the state and implemented by government officials. Donor-supported initiatives fall in this latter category: participatory spaces that are imposed from the outside (Mansuri and Rao 2013). x Contextual factors have attracted much attention in the literature, given findings of mixed outcomes of CE. The CE Strategic Framework contains a useful compilation grouped into demand-side factors (people’s willingness and capacity to engage); supply-side factors (politicians’ and officials’ willingness and capacity to respond to citizen voice and participation); and legal, economic, and political factors (history, power relations, legal frameworks, and so on). Inequality and the possibility for elite capture is often highlighted as a contextual factor that may cause negative outcomes. x Mansuri and Rao (2013) criticize the quality of implementation of many participatory and decentralization projects for paying insufficient attention to contextual factors and for weak monitoring and evaluation (M&E)—aspects that invariably will also be part of the scope of this evaluation. 7 17. CE activities and mechanisms, their goals, and the channels through which impacts are achieved vary by type of project. Results chains for CE are therefore specific to the area of analysis. The Strategic Framework identifies five broad “outcome areas” amenable to the inclusion of CE activities which this evaluation chose to collapse into four areas (figure 2): public service delivery; natural resource management; governance and public financial management; and social inclusion and empowerment.6 Specific CE activities are used in each of these areas to support and enhance the achievement of the primary development objectives. The intermediate and final results (last column) are specific to each area. By contrast, the same CE activity or mechanism may be used in more than one area. Figure 2: A Framework for the World Bank Group’s Approach to Citizens Engagement Area Tools and mechanisms Intermediate and final results Public service Accessibility of service delivery delivery Consultations Coverage of service delivery Quality of service delivery Natural resource Grievance Redress Mechanisms (GRM) Increasing participation of CSOs management Disclosure of contracts Increased revenue transparency Collaboration in decision-making More sustainable resource management More equitable distribution of resources Governance and Changes in policy, regulation, reforms public financial Citizen-led monitoring and evaluation or Improved transparency management oversight Reduced corruption More active citizen and community participation Improved responsiveness to citizen demand More inclusive budget processes Empowering citizens with resources Pro-poor fiscal policies Improved governance Social inclusion Building citizen capacity for engagement Inclusion of marginalized groups and empowerment Social cohesion Empowerment Context Willingness of and capacity to engage of citizens and governments. Sociopolitical, economic, cultural, legal, and other factors 6 A stocktaking done for the Strategic Framework found that CE activities are most frequently found in public service delivery, followed by natural resource management and social inclusion and empowerment, and finally by governance and public financial management. Based on their relatively lower frequency and some similarity in issues, this evaluation chose to group governance and public financial management into the same outcome area. 8 PREVIOUS IEG FINDINGS 18. The proposed evaluation is the first at IEG to focus explicitly on the World Bank Group’s approach to citizen engagement in a comprehensive way. Previous IEG evaluations have touched on related concepts such as consultations, stakeholder engagement, participatory methods, and community-driven development (CDD). IEG has considerable evidence and experience on the topic. Evaluations covering these themes include social funds (OED 2002); the Poverty Reduction Strategy Initiative ([PRSI], OED 2004); community-based and community-driven development (OED 2005); Poverty and Social Impact Analysis ([PSIA], IEG 2010a); safeguards (IEG 2010b); governance and corruption (IEG 2011); self-evaluation systems (IEG 2016a); and several project performance assessment reports (PPAR) on CDD projects (IEG 2014, 2015, 2016b). 19. The OED evaluation of social funds found that social funds have increased awareness of the potential of participatory approaches. It also highlighted the challenge of engaging all members of the community in the project’s activities and avoid elite capture, and recommended the adoption of measures to ensure that beneficiaries are adequately informed and consulted. 20. The OED evaluation of community-based and community-driven development also highlighted the increasing use of participatory approaches in a variety of sectors that involve local communities. It identified the lack of buy-in from borrower officials as a specific challenge. The evaluation found that concerns about capacity levels in communities and the unwillingness of officials to devolve control over decisions and resources to communities can undermine community engagement. 21. The evaluation of the Poverty Reduction Strategy Initiative found that, according to most stakeholders, the Poverty Reduction Strategy process allowed for increased transparency and involvement of new actors into the development dialogue. CSOs, however, did not perceive they had more influence over the design of Poverty Reduction Strategies. The evaluation found a lack of clear intermediate process indicators relating to the participatory requirement, which in turn led to lack of clarity and incompatible expectations among stakeholders. 22. The Poverty and Social Impact Analysis evaluation found wide variability in stakeholder participation in the examined PSIAs, with no consultations mentioned in about two-fifths of the sample. The identification of relevant stakeholders had been variable, leading to a moderate effect on country policies. 23. The safeguards evaluation helped inform the World Bank’s new Environmental and Social Framework, which was approved in August 2016 after three rounds of extensive global consultations and will take effect in 2018. Qualitative research conducted by IEG revealed the widespread perception among World Bank staff and NGOs that safeguards and sustainability policies generate multiple benefits, including enhanced citizen voice at the local level and greater citizen ownership at the level of civil society. 9 24. The evaluation of the World Bank Country-Level Engagement on Governance and Anticorruption (GAC) looked at the first GAC strategy (2007–2011) and found that World Bank shareholders and various CSOs have shaped the governance agenda. Some groups have increasingly voiced strongly held views that the World Bank should provide more, not less, support to governance. 25. The report on self-evaluation systems (ROSES) of the World Bank Group found that Implementation Completion and Results Reports (ICRs) did not systematically report on mandatory CE activities related to safeguards. Further, beneficiary surveys measuring satisfaction were used in less than half of projects with identifiable beneficiaries, with survey results not well integrated into the sampled ICRs. Half of the reviewed projects contained at least one citizen engagement indicator in the results framework. However, many indicators captured feedback at the end of the project cycle, which was too late to inform mid-course corrections. CE plays a marginal role in World Bank Group self-evaluation. 26. IEG recently adopted a more in-depth qualitative fieldwork methodology for its CDD PPARs to better capture the voice of project beneficiaries, a methodology that will be a useful starting point for fieldwork done for this evaluation. Findings from these in-depth PPARs align with the broader literature on citizen engagement presented above. For example, a PPAR on the Poverty Reduction Fund in Lao People’s Democratic Republic highlighted the importance of context in that there was incompatibility between the government’s centralized approach and broad and deep participation of local communities (IEG 2016b). A PPAR of the Fadama II project in Nigeria identified elite capture in participatory projects as group members with stronger interpersonal networks had more voice in the choice of and access to assets financed by the projects (IEG 2014).7 Purpose, Objective, and Audience 27. This proposed evaluation aims to inform the Board and Management of the World Bank Group on the effectiveness of World Bank Group citizen engagement activities to support development processes and outcomes. It will be completed by the target date (end of FY18) to integrate beneficiary feedback into 100 percent of Investment Project Financing operations financed with IBRD loans or IDA credits with identifiable beneficiaries. It is expected that the evaluation will feed into any review and update of corporate CE goals, targets, and approaches by management at that time. 28. This evaluation aims to support both of IEG’s overarching goals, namely to assess (i) the relevance, efficacy, and efficiency of the World Bank Group’s operational activities, and (ii) the implementation of its strategy and enable mid-course corrections. 7 IEG has also done other relevant evaluative work, including in several participatory PPARs, which the evaluation will draw on. 10 29. The stakeholders for this evaluation are the World Bank Group Board’s Committee on Development Effectiveness (CODE); management of the World Bank Group Institutions; the CE Secretariat; the GPSA; operational staff, especially those working on CE activities in the World Bank, IFC, and MIGA; borrower and implementing agencies collaborating with the World Bank Group on CE; select civil society organizations and academics; and staff of donor agencies. Evaluation Questions and Scope EVALUATION QUESTIONS 30. The overarching questions for this evaluation are “How effectively has the World Bank Group mainstreamed citizen engagement at the project, program, country, and corporate level, and what is the evidence on how this process contributes to the achievement of development outcomes?” This will be operationalized through the following descriptive and evaluative questions: 1. What has been the quality of design of CE activities? Is there attention to context factors such as the capacity and willingness of stakeholders to engage? When are project-specific mechanisms used to engage and when do efforts build on and strengthen existing national, sectoral, or regional platforms for participation? 2. What has been the quality of implementation of CE activities? How are CE activities implemented and monitored? Are expectations created at design met during implementation? What are the costs of CE, including for participants? Who participates and whose voices are heard (e.g., women, men, rich, poor)? 3. What are the objectives and results of CE? What is the World Bank Group’s rationale for mainstreaming CE? What does participation achieve, and does it help make projects and governments more responsive to citizen needs? Does it help improve intermediate and final outcomes? Are there unintended outcomes? 4. Is the corporate environment enabling of CE? Is there a clear approach that is understood and owned by staff? What are the incentives, motivations, resources, and constraints for staff to integrate and innovate on CE? Is there learning on what works for CE? Have tools, mechanisms, and approaches evolved over time? What is the role of the GPSA? What has been the impact of the Strategic Framework? These questions will be explored at the project, program, country, and corporate levels, as explained in the Evaluation Design section and in attachment 3. 31. IFC and MIGA stakeholder and community engagement activities will be assessed in relation to the effectiveness of the Performance Standards to achieve stakeholder engagement and IFC’s and MIGA’s commitments to environmental and social sustainability articulated in the Sustainability Framework, as well as in the context of country level work. 32. How the World Bank Group takes context into account when designing and implementing CE will be assessed. The Strategic Framework classifies contextual factors into three types: (i) the willingness and capacity of citizens to engage with governments and service 11 providers; (ii) the willingness and capacity of governments to engage with citizens and share information; and (iii) socioeconomic and legal factors that impact on the outcomes of CE. These factors include the degree of decentralization, history of citizen-government relationships, the rule of law, inequality and social inclusion, norms, values, and culture. 33. As discussed, the World Bank Group considers CE important for achieving better development outcomes and this will be the perspective adopted by this evaluation.8 The evaluation will also explore to what extent CE policies and programs have broader objectives, and results, such as inclusion of marginalized groups, empowerment, social cohesion, and social mobilization. Scope 34. The scope of this evaluation was defined in a consultative process that involved management and CSOs—taking as starting points the Strategic Framework, a stocktake of the history of CE in the World Bank, and published literature reviews. An earlier version of this Approach Paper was discussed with World Bank Group representatives at a workshop in December 2016. 35. The main challenge for scoping this evaluation consists in the diffused presence of CE activities across World Bank Group operations at all levels and for all types of instruments (table 1), stemming from the cross-cutting nature of CE. This makes it infeasible to identify a singular portfolio and requires selectivity in defining the scope. 36. Arguably, the evaluation can make its most strategic contributions though country-level perspectives. The country approach will therefore be a central element of the analysis. The country-level angle will review issues related to sustainability, commitment, coordination, ownership, and coherence, allowing a focus on how the CE approach works. In addition, implementation and results of individual activities and mechanisms will be reviewed. Countries will be selected based on a number of criteria including openness to CE, fragility, and diversity across regions and outcome areas (Appendix 3). 37. Since many CE activities are not exhaustively documented, traditional portfolio reviews will be de-emphasized. World Bank CE activities are tracked at the project level primarily for reporting on the corporate indicator. Based on these data, figure 3 shows how frequently different CE tools and mechanisms are being used by the World Bank. Mechanisms such as satisfaction surveys, GRMs, and consultations are the most commonly reported ones. The data also indicate a doubling of the number of reported indicators from FY15 to FY16. However, this data source only reports one indicator per project, even if more than one CE mechanism is deployed, resulting in a partial picture. 8 However, the evaluation recognizes that CE is also important in its own right, for example, to increase the sense of citizenship. 12 38. The selection of CE activities and mechanisms for a deep-dive analysis will be accomplished based on the country-level analysis. The country lending portfolios will be used as a starting point. This evaluation will then assess how selected CE activities are designed and implemented and review evidence of results. The evaluation will refer to area-specific results chains for specific pieces of the analysis as needed. Figure 3. Prevalence of Citizens Engagement Indicators in World Bank Projects Citizens' Engagement Indicators 120 100 80 Number 60 40 20 0 FY15 FY16 Source: Data provided by the CE Secretariat. Note: A number of indicators are provided, not mechanisms. A project can have more than one mechanism but it will report one indicator. The indicator is tracked for all Investment Project Financing (IPF) operations financed with IBRD loans or IDA credits. 39. The evaluation will look at financing instruments (investment, policy-based, Program for Results, IFC investments, MIGA guarantees); Technical Assistance; country approaches; the GPSA and, to a limited extent, other partnerships programs; and the corporate scorecard indicator. Referring back to table 1, it will cover the entry points listed there with the following exceptions: x CE in analytical work was addressed in IEG’s 2010 study of the Poverty and Social Impact Analysis and will not be systematically covered. The evaluation will however systematically analyze relevant ASA in the case studies. x Likewise, World Bank safeguards policies will not be evaluated as such, but the CE mechanisms that have been used to fulfill safeguards requirements will be assessed alongside the other CE mechanisms. x The annual CSO forum and corporate consultations, such as those done for the design of the Environmental and Social Framework, have different goals and audiences than operational CE, and are therefore excluded. CSO engagements will be addressed only to 13 the extent they are an avenue for, or have direct implications for, CE. Broad, multipurpose CSO consultations will not be reviewed. x The operations of the Inspection Panel and IFC’s and MIGA’s Compliance Advisor/Ombudsman related to the implementation of safeguards also are not directly relevant and will not be reviewed. 40. The focus will be on recent years while being cognizant of the long history of CE in the World Bank Group. The evaluation time frame will be flexibly geared to the questions. For example, analysis related to IFC and MIGA will cover the time period since their adoption of the PS, analysis related to CE mainstreaming processes will focus on the implications of adopting the Strategic Framework in 2014, and analysis of outcomes of CE will not be time bound. Evaluation Design 41. The complex nature of the topic will be best served by adopting a multi-level, mixed- methods design with both quantitative and qualitative methods, as laid out in Attachment 3 and Table 2. The evaluation will cover the topic at four levels—corporate, country, sector and outcome area, and mechanism. The bedrock will be 10–11 field-based country case studies and 2–3 desk-based ones. Cases will include fragile contexts as well as high and low willingness to engage citizens. Several complementary PPARs are planned and will help the team cover in some depth a sample of World Bank and IFC projects with objectives or components related to CE. Whenever possible, the PPARs will be conducted in the selected case countries so as to provide in-depth reviews of particular projects to complement the review of the entire country. Attachment 3 shows the tentative list of case studies. 42. Cases will be selected with a view to ensure coverage across the four outcome areas identified in the Strategic Framework (see Table 2): public service delivery; natural resource management, governance and public financial management; and social inclusion and empowerment. Stratification across these outcome areas will help ensure diversity in the types and goals of CE activities reviewed. 9 43. This evaluation will focus on the part of the results chain that the World Bank can be deemed accountable for—that is, attributable results in terms of changes in awareness, policy, practice, behavior, and power relations for better development. The evaluation will further use literature reviews to explore/describe how CE tools and mechanisms may be linked to final development outcomes, develop theories of change, and explore process tracing and other methods to link World Bank Group CE activities to intermediate and possibly final outcomes. The evaluation methodology will build on the literature, for example in relying on existing findings about what are key enabling or conditioning elements necessary for results, and review 9 A category of projects that frequently include CE mechanisms, often in response to safeguards, are infrastructure projects. These projects will be reviewed through various prisms such as access to transport, energy and other public services, as well as in relation to safeguards implementation. They will be included in structured reviews of CE mechanisms whenever relevant. 14 how well the design of World Bank Group CE activities factors in the presence or absence of those elements. Table 2. Main Goals of Each Component of the Evaluation Approach Evaluation Main goals approach/method Country sector case studies - Assess the quality of design of CE activities in a specific context; - Assess the quality of implementation of CE, the quality of monitoring, the cost effectiveness and the results achieved; - Understand who are the citizens, how they are engaged; - Assess when and how the World Bank Group strengthens and uses country systems and when and how it embeds specific CE mechanisms in its work; - Assess the critical factors determining meaningful engagement (or lack thereof). Structured review of IFC and - Assess IFC and MIGA’s oversight of client’s Performance Standards implementation MIGA operations during the portfolio relationship and other policy commitments regarding Stakeholder Engagement. Structured review of country - Assess how the World Bank Group accounted for context factors related to the strategies capacity and willingness of stakeholders and governments to engage with citizens. Structured literature review - Assess how the design and implementation of CE has impacted results of interventions in specific outcome areas; - Identify theories of change. Structured desk-based - Understand the extent to which the design of CE activities is informed by existing reviews of interventions and evidence of what works and for whom in a specific context; mechanisms - Assess how much and how effectively the WBG builds on country systems; - Assess the quality of design of CE mechanisms. PPARs - Assess quality of design, implementation, and results of CE in the context of specific projects. Big data analytics - Assess who are the citizens that engage and whose voices are heard; - Assess the alignment between citizens’ expectations and World Bank Group responsiveness. Analysis of corporate - Understand the corporate incentives, motivations and constraints for CE enabling environment - Understand how the World Bank Group learns and improves in CE. 44. Additional design elements will help the team generate broader findings that can extend and complement the case studies and PPARs. These include: structured reviews of the most salient CE mechanisms; a structured review of policy operations’ prior actions ; a structured review of CE in country strategies; assessment of the data process behind the corporate scorecard indicator used to monitor progress toward the CE target; literature reviews; interviews with teams, social and governance specialists, country management units, and country stakeholders; potentially an electronic survey; and a review of the Global Partnership for Social Accountability. The team will review IFC and MIGA’s implementation of the community engagement dimensions of the Performance Standards based on IEG’s environmental and social project performance reviews as well as select site visits. The evaluation will collaborate with the United Nations Global Pulse to assess how responsive to citizen’s voice the World Bank has been on social media (Twitter) in one country (this is referred to in the World Bank Group Strategy as “digital engagement”). The purpose will be to analyze the approach, the 15 implementation, the results, and extract interesting lessons. A traditional portfolio review covering all projects with elements of CE will not be possible. See Attachment 3 and Table 2 for additional details. Quality Assurance Process 45. This evaluation will go through IEG’s regular quality assurance process. The peer reviewers are Vijayendra Rao (lead economist, DEC); Cyprian Fisiy (former director, Social Development); Jonathan Fox (professor, American University); and Ritva Reinikka (former sector and country director, World Bank). In addition, the draft Approach Paper has been discussed at a workshop with management representatives and, likewise, the draft final report will be discussed at a workshop. The detailed methodology plan will be finalized in close consultation with IEG’s methods advisor. Expected Outputs, Outreach and Tracking 46. The main output will be an evaluation report of no more than 50 pages. This will be complemented by PPARs, working papers analyzing knowledge gaps, and potential learning engagements to be identified. 47. The evaluation team has already started and will continue to conduct active external outreach to civil society and to staff coordinating CE activities. The Civil Society Policy Forum during the 2016 Annual Meetings was used to engage with CSOs and other stakeholders and gather input and reactions to the evaluation. The team proposes to similarly engage at future forums. The team has and plans to maintain a constructive dialogue with staff working on CE. The final evaluation report will be disseminated both internally and externally. Resources 48. Under the direction and guidance of Marie Gaarder (Manager) and Auguste Kouame (Director), the evaluation will be prepared by a team from all parts of IEG and led by Elena Bardasi and Rasmus Heltberg and comprising Sahra Abdi Nur, Anna Aghumian, Eduardo Fernandez Maldonado, Gisela Garcia, Lauren Kelly, Gurkan Kuntasal, Mari Noelle Lantin Roquiz, Javier Lanza, Christopher David Nelson, Estelle Raimondo, Kathryn Steingraber, and Giovanni Tanzillo. Barbara Befani, Anis Dani, and Jos Vaessen will advise the team. Faby Chacaltana and Yezena Yimer will provide administrative support. 49. The evaluation will be prepared with an estimated net budget of $977,000 (including dissemination) of which 67 percent will be spent in FY17. The report will be finalized and submitted to CODE in the third quarter of FY18. 16 Attachment 1. References Amartya, Sen. 2005. The Argumentative Indian. London: Penguin. Gaventa, John, and Gregory Barrett. 2012. “ Mapping the Outcomes of Citizen Engagement.” World Development 40 (12): 2399–2410. IEG (Independent Evaluation Group). 2010a. Analyzing the Effects of Policy Reforms on the Poor: An Evaluation of the Effectiveness of World Bank Support to Poverty and Social Impact Analyses. Washington, DC: World Bank. ———. 2010b. Safeguards and Sustainability Policies in a Changing World: An Independent Evaluation of World Bank Group Experience. Washington, DC: World Bank. ———. 2011. World Bank Country-Level Engagement on Governance and Anticorruption: An Evaluation of the 2007 Strategy and Implementation Plan. Washington, DC: World Bank. ———. 2014b. Project Performance Assessment Report of the Second National Fadama Development Project. Washington, DC: World Bank. ———. 2015. Project Performance Assessment Report. Investing in Social Capital:͒Lessons from a Decade of Village Development in Sri Lanka. Washington, DC: World Bank. ———. 2016. Project Performance Assessment Report: Lao People’s Democratic Republic Poverty Reduction Fund Project. Washington, DC: World Bank. ———. 2016. Behind the Mirror: A Report on the Self-Evaluation Systems of the World Bank Group. Washington, DC: World Bank. Mansuri, Ghazala, and Vijayendra Rao. 2013. Localizing Development: Does Participation Work? Policy Research Report. Washington, DC: World Bank. OED (Operations Evaluation Department). 2002. Social Funds: Assessing Effectiveness. Washington, DC: World Bank. ———. 2004. The Poverty Reduction Strategy Initiative: An Independent Evaluation of the World Bank’s Support Through 2003. Washington, DC: World Bank. ———. 2005. The Effectiveness of World Bank Support for Community-Based and –Driven Development: An OED Evaluation. Washington, DC: World Bank. United Nations. 2008. People Matter: Civic Engagement in Public Governance. World Public Sector Report 2008. New York: United Nations. World Bank. 2003. Making Services Work for Poor People. World Development Report 2004. Washington, DC: World Bank. ———. 2014. Strategic Framework for Mainstreaming Citizen Engagement in World Bank Group Operations. Washington, DC: World Bank. https://openknowledge.worldbank.org/handle/10986/21113 World Bank Group. 2013. World Bank Group Strategy. Washington, DC: World Bank. 17 Attachment 2. Detailed Timeline AP to Management January 12, 2017 World Bank Group Management comments due February 3, 2017 E-submission to SEC of AP March, 2017 One Stop Meeting (Report) December 2017 REACT Workshop with counterparts January 2018 Send evaluation draft to Bank Mgt. January 2018 Report finalized and e-Submitted to CODE March 6, 2018 CODE discussion? TBD 18 Attachment 3: Methodological Approach The evaluation’s objective is to assess how effectively the World Bank Group has mainstreamed citizen engagement at the project, program, country, and corporate levels, and what is the evidence on how this process contributes to the achievement of development outcomes. These overarching questions are further articulated as several lines of inquiry summarized in table 3-3. The evaluation will adopt a multilevel, mixed-methods design. Given the particularly complex nature of the topic, it was decided that a country lens was to be the bedrock of the evaluation; sector specific portfolios will be analyzed for each country case study. The evaluation will cover the topic at four levels (country, sector, mechanism, and corporate) through a range of evaluation approaches, but with diverse degrees of depth and breadth. What follows is a description of the proposed evaluation approaches, recapitulated in tables 3-2 and 3-3, respectively, by evaluation questions and by levels of analysis. NESTED COUNTRY/SECTOR CASE STUDIES Given its time and budget constraints, the evaluation aims to conduct at least ten field-based and two desk-based case studies. The case selection strategy will be an iterative process, considering a number of conditions concomitantly: x Sufficient diversity of country context along the important dimensions of “country willingness to engage citizen” and fragility; x Representation of all six regions and of IDA and IBRD countries; x Diversity of substantive CE activities across outcome areas; x Countries highlighted by key informants as having prominent CE programs going back some time, as well as countries considered to not have that. The case identification processes will involve a number iterative steps: x Countries will be grouped according to their “willingness” to engage with citizens. The “Voice and Accountability” indicator within the World Governance Index will be used as proxy. x The World Bank, African Development Bank, and Asian Development Bank harmonized list of states in a fragile situation will be used in addition to considerations of fragile situation within otherwise nonfragile states. x Conversations with key informants knowledgeable about CE will help identify countries with prominent CE programs; the duration of those programs; and the likelihood of obtaining good data on results. Suggestions from key informants will be cross-referenced with a list of the size of the World Bank portfolio in the core outcome areas (public service delivery, public finance management, natural resource management, governance, and social inclusion) with a view to achieve diversity of cases across outcome areas. x Pragmatic considerations will also factor in. For example, some countries and issues have been extensively studied already (India, Indonesia, CDD) and can be covered by desk review. Opportunities to combine case studies with PPAR work, and to visit countries where relatively few IEG missions have visited in recent years will also be explored. 19 A tentative list of case studies, subject to budget, ability to travel, and evaluative judgments, is shown in Table 3-1. Table 3-1. Indicative list of case studies Region Type of study Fragile Voice and context accountability percentile rank Burkina Faso AFR Impact evaluation 40 Cambodia EAP Desk study, PPAR 19 Dominican Republic LAC Country study 51 Ethiopia AFR Country study 14 Kazakhstan ECA Country study 17 Kyrgyz Republic ECA PPAR, Country study 32 Lao PDR EAP Country study, PPAR 4 Madagascar AFR Country study Yes 34 Mali AFR Country study, PPAR Yes 40 Morocco MENA Country study 28 Nepal SAR Desk study, PPAR 33 Pakistan SAR Country study, PPAR 27 Peru LAC Country study, IFC PPAR 54 Philippines EAP Country study, PPARs (One region) 52 Vietnam EAP PPAR 11 West Bank and Gaza MENA Country study Yes 23 Note: Source for the “Voice and Accountability percentile rank” is the Worldwide Governance Indicator. Values are for 2015. THREE NESTED LEVELS OF ASSESSMENT In each case study selected for field-based inquiry, the World Bank Group’s support to citizen engagement will be assessed at three levels: country, sector or outcome area, and project. x Country: context; government priorities; role, goals, strategy, portfolio, and results of the World Bank Group; the extent to which the World Bank Group supports and builds on domestic frameworks for CE; synergies across operations. x Sector or outcome area: the quality of design, implementation, and results of CE activities for the portfolio of activities in one or two selected sectors. x Project and CE mechanism: in a subset of cases, in-depth inquiry into CE mechanisms deployed in specific projects will take place. Data collection and analysis A structured protocol for case studies will be developed. For field-based inquiry a core evaluation team will conduct early on a pilot study where all of the data collection tools will be 20 tested and calibrated. The method will be finalized and deployed in the other case studies. The data collection methods will comprise the following: x Desk-review of project documents and country strategies x Literature review of relevant CE literature in the country x Interviews of Bank staff in Country Management Units x Interviews with teams x Interviews with government officials x Interviews with project management units x Interviews with CSOs x Interviews and focus groups with citizens/beneficiaries/stakeholders (whenever appropriate and feasible). For each country, a case narrative will be drafted as a basis for synthesizing evidence in the overall evaluation report. In addition, specific case-based analytical approaches are under consideration, such as: x Cross-case comparison: for each type of outcome area, evidence from different countries will be compared and contrasted. x Within-case inquiry: Given the importance of understanding the processes underlying citizen engagement, specific within-case methods, such as process-tracing will be explored x Stakeholder mapping: Given the centrality of understanding who gets to engage, specific stakeholder mapping techniques will be used, including possibly Social Network Analysis. STRUCTURED REVIEW OF COUNTRY STRATEGIES In order to assess the extent to which the World Bank Group has mainstreamed CE into its country engagement strategy a structured reviews of country strategy documents and process will be undertaken and will build on the existing analysis of 22 SCD/CPF recently undertaken for the FY17 IEG evaluation. Document reviews and in-person or phone interviews will serve as a basis for the assessment. STRUCTURED LITERATURE REVIEW For the purpose of this evaluation, IEG will review the evidence on CE implementation and impact in four outcome areas: (i) public service delivery; (ii) social inclusion and empowerment; (iii) natural resource management; and (iv) governance and public financial management. For each outcome area, the review will attempt to shed light on the following questions: 1. What are the CE mechanisms used? 2. What are the objectives of the CE mechanisms? What results are the CE mechanism trying to achieve and how are they supposed to contribute to improving the outcomes of the development intervention? 21 3. Is there evidence that the CE mechanisms are implemented as planned (i.e., implementation fidelity)? For instance, is there evidence that the feedback loops were closed? Is there a discussion of the incentives faced by the implementers (e.g., “street level bureaucrats”) of the CE mechanism? 4. How is the success of CE measured? Two critical questions: What is the indicator used to measure the outcome (i.e., measurement validity)? What is the evaluation design used to measure plausible causality? 5. What factors affect the implementation and success of the CE mechanism? The possibility of synthesizing the findings from the literature through a Qualitative Comparative Analysis to identify patterns of regularity in particular causal configurations —of specific citizen engagement mechanisms leading to particular outcomes under certain contextual circumstances—will be explored. These causal packages, which are well supported by empirical evidence in the literature, will also serve as a basis to assess the extent to which World Bank Group CE activities are likely to lead to positive results. STRUCTURED DESK-BASED REVIEWS OF INTERVENTIONS AND MECHANISMS x Systematic review of a random sample of investment operations approved after 2014, assessing (1) the CE mechanisms described in the Appraisal document and (2) reported on in the supervision report. Possibly, a subset of projects including select mechanisms will be identified for more in-depth analysis. x Structured review of policy operations’ prior actions: Through Development Policy Financing (DPF), the World Bank also supports or requires CE under certain circumstances. The evaluation will use the database built for the review of social and environmental risk management in DPF to conduct a structured review to assess CE prior actions. SPECIFIC REVIEWS OF IFC AND MIGA The team will review IFC and MIGA’s activities to overseeing clients’ implementation of the community engagement dimensions of the Performance Standards based on IEG’s environmental and social project performance reviews. All projects reviewed by IEG during the last three years will form part of the review. IEG will develop a systematic coding scheme focused on the design and implementation of stakeholder engagement by clients, evidence on outputs and intermediate outcomes, the degree to which clients internalize CE mechanisms, and GRMs. Documentation quality will also be assessed. This data will be analyzed for trends over time, across regions and sectors, and for countries with strong and weak legal requirements. The assessment will be complemented by site visits to select projects that benefit from IFC support to review how they pass on the Performance Standards requirements to subprojects and by interviews with IFC and MIGA social specialists. 22 PPARS Several complementary PPARs are planned and will help the team cover in some depth a sample of Bank-supported projects whose objectives relate to CE. Because they can be staggered, these PPARs will also allow the team to gradually refine its methods and expertize. These will cover CDD projects in Kyrgyz Republic, Mali, and Nepal; a Demand for Good Governance project in Cambodia; community-based water in Nepal; community-based forestry in Lao PDR and Vietnam; IFC advisory in Peru; and possibly more. IEG’s ongoing impact evaluation of a citizen scorecards project in Burkina Faso will also be an input. BIG DATA ANALYTICS OF WORLD BANK’S DIGITAL ENGAGEMENT Given the rising importance of social media as a platform for citizen’s voice, the evaluation will also seek to assess how engaged with and responsive to citizen’s voice on social media the World Bank has been. To do so, the evaluation team is attempting to partner with United Nations Global Pulse, a leading lab on the use of big data analytics in international development to leverage Global Pulse’s access to Twitter database. This experimental evaluation approach will be tested in one specific country where the rate of Twitter penetration and World Bank Twitter activity is quite high. FOCUSED SURVEYS AND KEY INFORMANT INTERVIEWS A series of interviews and, possibly, an electronic survey of selected staff is being planned. These will aim to explore staff’s attitudes, knowledge and resources regarding CE. Selected staff includes country directors, country managers, other CMU staff, social development specialists, and teams in various sectors. (Interviews with external stakeholders will also be done as part of country case studies.) STRUCTURED REVIEW OF THE GPSA AND OTHER RELEVANT PROGRAMS AND PARTNERSHIP Global Partnership for Social Accountability will be assessed as an in-depth case study to inform the evaluation. The partnership review will be carried out by adapting IEG’s evaluation framework for assessing global and regional partnership programs. This will be based on a desk review of key program documents, GPSA’s external evaluation(s), and interviews with the program stakeholders and beneficiaries. Where possible, country visits for the main evaluation will be used to provide evidence on program’s contribution. External evaluations and interviews will be used by IEG to get insights on the relevance and effectiveness of other partnership programs, such as the Open Government Partnership, and the activities funded by trust funds, such as the Japan Social Development Fund. REVIEW OF CORPORATE INDICATORS The strengths and limitations of the corporate scorecard indicator used to monitor progress toward the CE target will be assessed by reviewing how the indicator data is collected, aggregated, and used. 23 Causal inference and generalizability of findings Many of the methods proposed rely on configurational causation, such as qualitative comparative analysis, process tracing, and other methods of causal inference. These techniques – combined with structured literature reviews for each main outcome areas – will allow reconstructing CE activity-specific results chains and establishing whether the intervention (or CE mechanisms) made a difference, for whom, and under what circumstances. The evaluation cannot rely on a global portfolio review and comprehensive assessment of the universe of corporate CE activities. The design of different evaluation elements will aim at optimizing the ability to draw causal inferences across cases spanning diverse contexts and outcome areas. The nested country-sector approach will allow for a certain degree of generalization depending on the quality of the selection of countries, sectors, and activities as well as on the degree to which there is convergence or divergence in patterns uncovered in these cases. The evaluation will aim for generalizability of findings on other evaluative questions, such as the quality of design, through survey and structured review techniques. 24 Table 3-2. Evaluation Design by Evaluation Questions Nested Literature review CE PPAR Structured Key informant Big data country/sector mechanisms desk- interviews and social case studies case studies based network reviews analysis What has been the quality of 9 9 9 9 9 design of CE activities? x How has the World Bank Structured Group accounted for context reviews of factors such as the capacity country X Identify influential context and willingness of x strategies factors stakeholders and (building governments to engage? on IEG FY17) x To what extent is the design Structured of CE activities informed by reviews of existing evidence of what Identify configuration of PAD and x x works, for whom and in what Context/mechanism/outcome ICRR of context? “core CE projects” x How much and how Structured effectively does the World reviews of Bank Group build on and X x X DPOs prior strengthen countries’ own actions systems for CE? x Are potential synergies across projects exploited, X X especially at the country level? What has been the quality of implementation of CE 9 9 9 9 9 activities? x What are the incentives, motivations, and constraints X X X X for implementers on the ground? 25 x How are CE activities Structured monitored? x X reviews of ICRR x Is attention paid to the cost effectiveness of instruments x x x (including for citizens)? x Does the World Bank Group work with the right partners x x x x and intermediaries to achieve its CE objectives? x Who participates and whose voices are heard (e.g., x x x x women, men, rich, poor)? What are the objectives and 9 9 9 9 results of CE? x What is the World Bank Group’s rationale for mainstreaming CE? x Does CE help make World Bank Group projects and x x x x strategies more responsive to citizen needs? x Under what circumstances does World Bank Group’s CE help make governments x x more responsive to citizen needs? x Does CE improve intermediate and final x x x outcomes? x What are the unintended x x x outcomes? Is the corporate environment 9 9 9 9 9 enabling of CE? x Is CE understood and owned x x by Bank Group staff? 26 x What are the incentives, motivations, and constraints for task team leaders to integrate and innovate on CE? x What has been the impact of the Strategic Framework and x x of the corporate indicators? x Is there learning on what works for CE? How have tools, mechanism, x x x x approaches, and technology evolved over time? x What is the role of the Structured GPSA? review of GPSA 27 Table 3-2. Evaluation Design by Level of Analysis Country level Sector level Mechanism level Corporate level Country case X (X) studies Literature review X X Mechanism case X X studies Process Tracing X Key informant X X X X interviews PPAR X X X Structured desk X X X X reviews Social network X analysis Big data analytics X X QUALITY ASSURANCE MECHANISMS Steps will be taken to ensure the validity of findings and guarantee consistency of approach across members within the evaluation team. For instance, a case study template and interview protocols will be developed to ensure a common framework and evaluative lens across studies. Furthermore, the team will apply triangulation at multiple levels. First, triangulation will be applied by cross-checking sources of evidence within a given methodological component. For instance, within case studies, evidence stemming from interviews with CSOs, development partners, Bank staff, and citizens on the same topic will be compared and contrasted. Second, triangulation will be applied across evaluation components. For example, findings stemming from case studies will be cross-validated with findings emerging from surveys and structured document reviews. The evaluation team will also apply external validation mechanisms at various intervals during the evaluation process. For example, peer-reviewers and senior advisors will provide feedback at the beginning, during, and at the end of the evaluation process. A workshop was held with stakeholders at the inception of the evaluation process to validate the scope and the approach, and another will be convened at the end around the relevance and feasibility of the evaluation’s recommendations. 28