A F R I CA O P E R AT I O N S S E RV I C E S S E R I E S 75985 Enhancing Transparency and Accountability through Citizen Feedback Lessons from the E-ISR+ Pilot ore Operatio C ns S ervices Quality OS DE P Capacity Results C FM ASP EN THE AFRICA REGION | THE WORLD BANK GROUP AFRICA OPERATIONS SERVICES SERIES Enhancing Transparency and Accountability through Citizen Feedback Lessons from the E-ISR+ Pilot This note was developed by a core team sponsored by AFTOS, under the direction of Sahr Kpundeh, Adviser, Governance and Anti-Corruption. Members of the core team included Shanthi Kalathil (principle author) and Regina Wilson. This note draws heav- ily on the work of staff from across the Bank who have contributed to the Bank’s knowledge about E-ISR+ and social account- ability, including Mohammed Sidie Sheriff, Gibril Jalloh, Margaux Hall, Gabriel Dedu, Ko� Tsikata, Smile Kwawukume, Raymond Muhula, Michael Nyumah Sahr, Caroline Sage, Diane Zovhigian, Paula Rossiasco, Bronwyn Grieve, Ilke Corinne Ilgun, Louise Engulu, Milaine Rossanaly, Kate Bridges, Gael Raballand, and others who have worked on E-ISR+ across Africa. AFRICA OPERATIONS SERVICES SERIES Contents Abbreviations and Acronyms iv Preface v I. Background: Adapting to change 1 iii iii II. Objectives of E-ISR+: Feedback, transparency, accountability 3 III. E-ISR+ in Country Context 5 Social accountability 5 Third-party monitoring 6 Participatory monitoring and evaluation 6 Summarizing E-ISR+ in context 7 IV. Early experience with E-ISR+ 9 General project-level �ndings from initial E-ISR+ rollout 9 Methodology for implementing E-ISR+ 10 E-ISR+ experience in select African countries 11 E-ISR experience in Nigeria 11 E-ISR+ experience in Zambia 11 E-ISR+ experience in Sierra Leone 12 E-ISR+ experience in Ghana 12 E-ISR+ experience in Burkina Faso 12 E-ISR+ experience in DRC and Congo-Brazzaville 13 V. Lessons Learned from E-ISR+ Initial Phase 15 Aspects of E-ISR+ That Worked Well 15 Aspects of E-ISR+ That Presented Challenges 17 VI. Emerging Practices for E-ISR+ 21 References 25 ENHANCING TRANSPARENCY & ACCOUNTABILITY THROUGH CITIZEN FEEDBACK Abbreviations and Acronyms AFR Africa Region CDD Community-driven development CMU Country management unit CSO Civil society organization iv E-ISR+ External implementation Status and Results Plus GAC Governance and anti-corruption ISR Implementation Status and Results (Report) M&E Monitoring and evaluation NGO Nongovernmental organization TTL Task team leader AFRICA OPERATIONS SERVICES SERIES Preface The External Implementation Status and Results Plus (E-ISR+) system is a feedback, transparency, and accountability tool for the World Bank. The Implementation Status and Results Report (ISR), instituted in 2005, is an internal reporting tool on project implementation progress. In 2010, the ISR process incorporated E-ISR+ as a new pilot effort, designed to incorporate the views of bene�ciaries and bring key stakeholders and civil society into the project monitoring and evalu- ation (M&E) process. v E-ISR+ is intended to disclose current project information to external stakeholders, to obtain feedback from non-state players on project progress and results, and to systematically reflect external feedback in implementation reporting. In doing so, it incorporates concepts from social accountability, third-party monitoring, and participatory monitoring and evaluation to emphasize increased transparency, accountability, and stakeholder involvement in World Bank projects. E-ISR+, initially piloted in several countries in Africa, has been spearheaded by the Africa Region’s GAC-in-Operations team of the World Bank. The lessons learned from the initial roll- out help illuminate evolving practices as well as challenging roadblocks with respect to E-ISR+. Civil society organizations (CSOs), for example, build capacity in �nding and analyzing infor- mation, as well as the technical aspects of M&E projects. This third-party feedback highlights and addresses ongoing project challenges, sometimes leading to new governance-related initia- tives. E-ISR+ helps increase trust between civil society organizations and government and leads to a better environment for accountability and transparency in country. On the other hand, there are challenging concerns about conflicts of interest, CSO capacity, and getting buy-in from all parties. E-ISR+ is an important new feedback, transparency, and accountability tool for the World Bank. This report explains the purpose and relevance of the E-ISR+, examines ongoing efforts and cur- rent lessons learned, and discusses evolving practices for implementing E-ISR+ going forward. AFRICA OPERATIONS SERVICES SERIES I. Background: Adapting to change The Implementation Status and Results Reports (ISR), instituted in 2005, is an internal reporting tool on project implementation progress in World Bank operations. Since its origin, concepts such as accountability and participation have acquired more relevance within development 1 policy circles. In particular, the idea of “open development� has gained currency among major donors, reflecting an emphasis on transparency, responsiveness, access to information, citizen engagement, and community ownership of development solutions. In this sense, the concept of open development dovetails with the rise to prominence of social accountability mechanisms in development, which emphasize many of the same concepts. In 2010, ISR practice was adapted to reflect the World Bank’s shifting focus on increased trans- parency and accountability in Bank operations. The main sections of the ISR became accessible to the public, reflecting the Bank’s new access to information policies and an effort to open up more information about Bank operations to the external public. Moreover, reflecting the overall trend toward more open development, the ISR process tests a new pilot effort in several African countries. The design of the new process incorporates the views of bene�ciaries and brings key stakeholders and civil society into the project monitoring and evaluation (M&E) process. This new effort, External Implementation Status and Results Report Plus (E-ISR+), gets its start in 8 countries in the Africa Region (AFR). The AFR GAC-in-Operations team at the Bank spearheads the E-ISR+ effort. AFRICA OPERATIONS SERVICES SERIES II. Objectives of E-ISR+: Feedback, transparency, accountability 3 E-ISR+ is intended primarily to disclose current project information to external stakeholders, to obtain feedback from non-state players on project progress and results, and to systematically reflect external feedback in implementation reporting. Because the tool’s impetus is situated within the broader move toward open development, E-ISR+ also has broader objectives. These include strengthening the overall monitoring and effectiveness of government-implemented projects, improving accountability in World Bank M&E practices, and enhancing the Bank’s engagement with civil society as part of its larger social accountability agenda. If implemented correctly, E-ISR+ has the potential of delivering immeasurable bene�ts. One example is increasing the capacity of local bene�ciaries and stakeholders to self-monitor project progress. In traditional monitoring and evaluation, the intended bene�ciaries have little input into the process of determining whether or not a project ful�lls its objectives. Strengthening the capacity of bene�ciaries to think critically about monitoring and evaluation strengthens the overall project implementation. Another potential bene�t is the ongoing calibration of project progress and feedback in align- ment with local context, local conditions, and independent local sources of knowledge and expertise. By incorporating third-party input, local experiences and on-the-ground conditions, the Bank has a tool for crosschecking assumptions about project goals and objectives. Over time, this can lead to more accurate goal-setting and improved results. The potential of E-ISR+ also factors in increasing transparency and accountability of World Bank projects. E-ISR+ is designed to contribute to an improved environment for accountability gener- ally within the host country and particularly where civil society monitoring strengthens the abil- ity to hold government and other institutions answerable for their expected roles in a project. An improved environment for transparency and accountability is a long-term goal; it is also dependent on variable factors. The capacity building that takes place potentially in E-ISR+ can help contribute to broader accountability. AFRICA OPERATIONS SERVICES SERIES III. E-ISR+ in country context To better understand the potential of what can be achieved within a country context, this chapter briefly situates E-ISR+ within three complementary and, at times, overlapping, streams of practice—social accountability, third-party monitoring, and participatory monitoring and 5 evaluation. Social accountability Social accountability refers to an approach toward building accountability that relies on civic engagement—largely, a bottom-up phenomenon—in which grassroots stakeholders, civil society organizations, and the public directly or indirectly provide the impetus for demanding account- ability. The World Bank’s interest in social accountability stems from its recognition that such approaches have become fundamental to its core goals of promoting poverty reduction and sustainable development (World Bank 2006). The Bank recognizes three main arguments under- lying the importance of social accountability: governance, increased development effectiveness, and empowerment. The importance of social accountability initiatives is explicitly discussed in the World Development Report 2004, which highlights the importance of bringing accountabil- ity to service delivery (World Bank 2006). Social accountability mechanisms include participatory budgeting; public expenditure track- ing; and building an independent media sector, public commissions, and advisory boards. Such mechanisms are meant to complement rather than replace more conventional forms of account- ability such as elections, auditing systems, and administrative rules. Social accountability mech- anisms can emphasize either the “supply� side of governance by improving government capacity to deliver services or the “demand� side of governance by improving civil society’s capacity to access and analyze information and subsequently demand better government-provided services. Successful initiatives should include an emphasis on access to and use of information, build- ing civil society and state capacities and the synergies between them. They are most effective when institutionalized and linked to existing governance structures and service delivery systems (Malena et al. 2004). Increasingly, governments, civil society, and donors believe that social accountability can be particularly effective in service delivery, by holding both policymakers and service providers accountable. Such improvements in service delivery can serve as a toehold for further expansion of good governance, particularly in countries that do not feature a broader political environment that favors good governance. Civil society organizations (CSOs) are particularly important to the delivery of social services. They help inform citizens about their rights and entitlements, perform third-party monitoring through social audits, and conduct independent analysis (Ringold and others 2012). Social accountability measures are also linked to monitoring and evaluation of development projects. Pressure on governments (and donors) to focus on concrete results reinforces the ENHANCING TRANSPARENCY & ACCOUNTABILITY THROUGH CITIZEN FEEDBACK importance of improving M&E systems, particularly in contexts of weak accountability. New, results-based approaches often call for a stronger role for citizens in monitoring and evaluation to ensure that results are achieved. Such approaches take the grassroots-inspired, social account- ability-type emphases on bottom-up accountability and civil society capacity and combine it with traditional monitoring and evaluation of development projects. Third-party monitoring As its name suggests, third-party monitoring refers to a form of monitoring and evaluation that employs an independent third party to monitor project progress or report on results. While it can be used as an external accountability mechanism (to ensure that reporting of results is not biased), it can also be used as a form of strengthening stakeholder engagement. 6 Advocates of third-party monitoring, performing as form of stakeholder engagement, note that it can increase project transparency and promote trust between the project and its key stakehold- ers. The project being monitored, for instance, bene�ts by receiving an objective assessment of performance from outside observers, which from an external-relations standpoint can help address criticism and potentially strengthen support among key stakeholders (IFC 2007). Moreover, the presence of an external monitor not only increases the project’s accountability to all parties, but it can help boost the perceived credibility of the monitoring results on the part of the affected community and other potential bene�ciaries. For these groups, it can serve as an independent and objective source of information and reporting on the project (IFC 2007). Third-party participants in monitoring include civil society organizations, government regulators, academics, scientists, community representatives, technical experts, or community leaders from the private or not-for-pro�t sectors. Third-party monitoring can, and frequently does, include aspects of participatory monitoring and evaluation, such as stakeholder engagement. However, unlike participatory monitoring and evaluation, it does not necessarily contain an explicit focus on stakeholder engagement and capacity building. Participatory monitoring and evaluation Participatory monitoring and evaluation is increasingly common in development, particularly as the concepts of open development become more �rmly rooted in everyday practice. It differs though from conventional monitoring and evaluation. Participatory monitoring and evaluation gets its evolutionary start partly as a response to the criticism of conventional practices. Conventional monitoring and evaluation is primarily focused on measuring ongoing progress and judging the extent to which the project meets its stated objectives. During the process of con- ventional monitoring and evaluation, an evaluator’s objectivity and the literal distance between evaluator and subject is paramount. The overarching goal of conventional monitoring and evalu- ation is judgment of results rather than empowerment of stakeholders—that is, judging the out- puts, outcomes, and impact of the project in question (Estrella and Gaventa 1997). To gauge these elements, quantitative indicators are frequently employed. Over the years, criticisms emerge over this traditional approach. Critics generally assert that con- ventional monitoring and evaluation is costly; that it is often ineffective; that it does not actively involve project bene�ciaries and in fact even alienates them; and that its emphasis on largely quantitative measures ignores useful qualitative input (Estrella and Gaventa 1997). AFRICA OPERATIONS SERVICES SERIES While participatory monitoring and evaluation also focuses on impact assessment and project management and planning, its guiding principles are participation, learning, negotiation, and flexibility. Participatory monitoring and evaluation tends to place more emphasis on enhanced bene�ciary participation rather than funder-centered assessment of results. Rather than quantita- tive data gathering, it stresses recognition of locally relevant processes for gathering and using information and the context in which this occurs. An overarching goal of participatory monitoring and evaluation is stakeholder empowerment and M&E capacity building, which is believed to lead to increased self-reliance in project imple- mentation. Rather than taking place on �xed schedules (as is the case with conventional moni- toring and evaluation), participatory monitoring and evaluation can be frequent, less formal, and smaller in scale (Estrella and Gaventa 1997). 7 Summarizing E-ISR+ in context Taken in the context of these three separate but related concepts, E-ISR+ addresses several (if not all) aspects of each. Keeping the goals of social accountability, third-party monitoring, and par- ticipatory monitoring and evaluation in mind, E-ISR+ seeks to help better monitor government- implemented projects in order to improve development effectiveness. This is a profound bene�t to the country government and the World Bank. E-ISR+ also seeks to provide civil society with greater opportunities and strengthened capacity to monitor project performance, thus enhancing community ownership. In doing so, E-ISR+ can potentially bring an added layer of transparency, accountability, local ownership, and stakeholder participation to ongoing Bank operations. AFRICA OPERATIONS SERVICES SERIES IV. Early experience with E-ISR+ An early version of E-ISR was initially piloted by the World Bank in �ve African countries—Burkina Faso, Ghana, Kenya, Nigeria, and Zambia. The �ve countries were selected on the basis of demand for such initiatives from the Bank country of�ces. After the pilot phase, the E-ISR 9 process was conducted again (and renamed E-ISR+) and expanded to several other countries in Africa on a rolling basis. During this initial phase of the E-ISR+ process, each country management unit was free to determine for itself how best to address questions about methodology, such as how to contract out the assessment work. This chapter briefly describes general �ndings from the piloted E-ISR+ and the methodology employed, and some speci�c �ndings from a sampling of countries. General project-level findings from initial E-ISR+ rollout The project-level �ndings are indicative of the views of stakeholders across the projects assessed. These do not necessarily relate to the overall outcome or impact of the E-ISR+ process itself. These higher-level impact questions will be addressed in Chapter 5. There appears to be a general awareness and understanding of project objectives among bene�- ciaries and stakeholders although there are some notable examples where awareness of project objectives is low. There are varying levels of satisfaction among stakeholders on the quality of consultations before and during project implementation. While efforts aim to further project transparency, not all bene�ciaries have essential access to information. For instance, in a CDD project in Nigeria, �nancial information is displayed on a notice board for affected communities; however, bene�ciaries with limited �nancial literacy are unable to comprehend this unfamiliar terminology. Another example where positive action helps to overcome the problem is Burkina’s Additional Financing to the Transport Sector Project that seeks to involve local populations in roadwork through a Labor-Intense Road Works Program (HIMO). The feedback reveals that although bene�ciaries are aware of the program, many are unaware that they would be paid for their work on the road project. Many of these potential ben- e�ciaries accordingly expressed reluctance to get involved in the project. As a result, the project coordination unit takes necessary action to better inform populations on the HIMO method and get the required bene�ciary buy-in. Although progress is made on several projects, stakeholders often identify obstacles to this prog- ress, including delays in fund disbursement; lack of clear communication among project man- agers, government, and bene�ciaries; and lack of understanding by community leaders of the ENHANCING TRANSPARENCY & ACCOUNTABILITY THROUGH CITIZEN FEEDBACK potential negative impacts of projects. In Sierra Leone, for example, community leaders affected by a road-building project are not always well informed about the pre-construction phase, lead- ing to farmers not being compensated for destruction of property. Generally, target communities and other stakeholders express a desire to be more involved in the design phase of the project, and request both informal and formal mechanisms to stay apprised of project progress, communicate concerns, and report on potential incidents of corruption if necessary. In the Zambia Malaria Booster Project, for instance, project stakeholders suggest put- ting in place a monitoring system to ensure the community actually bene�ts from the project, as well as a system to encourage reporting suspicion of corruption with respect to selling bednets or diluting and re-selling pesticide sprays. 10 Methodology for implementing E-ISR+ The AFR GAC-In-Operations team allows flexibility for country teams to determine the best implementing strategy for E-ISR+. This flexibility allows for country teams to derive a number of lessons about the strengths and drawbacks of different approaches to methodology (such as the decision about whether or not to work with civil society), many of which are discussed in subsequent sections. Within the assessment teams themselves, methodology is not as varied. Assessment teams gener- ally tend to use a combination of in-depth key interviews and focus groups with project bene�- ciaries and other stakeholders to gather information. The majority relies on qualitative rather than quantitative inputs in their assessments. With respect to contracting out the work to the assessment team itself, methodology among Bank country management units (CMUs) varies. Many management units want to outsource the task to a professional organization that upholds the project objectives and employs quality control mea- sures. By necessity, this rules out incorporating potential bene�ciaries into assessment teams. Of three country reports completed by January 2012, one is by a market research �rm, one is by a civil society consultative group that had experience working with the World Bank, and one is by an individual in-country consultant with civil society experience. In addition, earlier pilots used other methods to contract out the M&E work. In the Ghana pilot, the CMU hired an NGO coali- tion to coordinate the work of other NGOs collecting information, based on guidance from task team leaders (TTL). In the Burkina Faso pilot, the pilot projects engaged selected members from civil society organizations to join the mid-term review mission of each project, and to produce an independent report on project performance. Each of the data-collection methods has bene�ts and drawbacks. Employing an outside profes- sional market research �rm (such as a survey company) clearly enables greater objectivity and quality control over the �nal E-ISR+ product and report. For conventional monitoring and evalu- ation, this represents a prudent choice. However, because the objectives of E-ISR+ are somewhat broader than simply monitoring project progress, contracting with a professional market research company does have some drawbacks in the context of E-ISR+. Employing a professional �rm does not necessarily increase the capacity of civil society groups, much less project bene�ciaries or affected communities, for accessing information about government or Bank projects, monitor- ing them, and holding institutions accountable. Employing a civil society group with no direct links to the projects is inherently a potential ben- e�t of the E-ISR+ process. For instance, it is likely to enhance the capacity of the civil society AFRICA OPERATIONS SERVICES SERIES organization being contracted to access government and Bank information, analyze data, adhere to professional M&E processes, and hold the government and the Bank accountable. Employing an individual consultant with civil society experience may potentially enhance the general capacity of civil society to monitor projects and hold institutions accountable, but only if the individual consultant is able to impart such knowledge to civil society colleagues. This may be an attractive choice for country of�ces since it is generally easier for the country management unit to contract with an individual than a civil society organization, highlighting the need for procurement support to country teams working on this initiative. E-ISR+ experience in select African countries 11 In this section, the E-ISR+ experience in some pilot and post-pilot countries is briefly described, drawn largely from country reports (which were not mandatory). E-ISR experience in Nigeria The E-ISR+ in Nigeria includes third-party monitoring of four World Bank-assisted projects: the Community and Social Development Project (CSDP), the Third National Fadama Development Project (FADAMA III), the Malaria Control Booster Project, and the Commercial Agriculture Development Project (CADP). The Bank engaged Nigeria’s Civil Society Consultative Group to carry out the third-party monitoring. The Consultative Group assessed community participation and decision-making across the projects, bene�ciaries’ satisfaction with project implementation and service delivery, and project outputs and outcomes. While full scope of the assessment across the four projects is not fully elaborated on here, there are some notable �ndings. For instance, the assessment team reports that while implementing agencies are making considerable efforts to communicate and share information with bene�cia- ries, ongoing information and communication gaps remain disengaged between the project and its bene�ciaries. The reports also �nd high rates of satisfaction and participation in some cases but less in others. Although decision-making tends to be inclusive (with efforts to integrate youth and women), access to decision-making for some in the community is inequitable and levels of local government support for some sub-projects are insuf�cient. In some cases, the team �nds low awareness regarding speci�c initiatives (like malaria bednets) and only passive participation of community bene�ciaries in malaria intervention and decision-making. The team concludes that there is need to sustain third-party monitoring. E-ISR+ experience in Zambia The E-ISR+ in Zambia covers third-party monitoring of seven World Bank-assisted projects: the Copperbelt Project, the Malaria Booster Project, the Roads Rehabilitation Project, the Public Service Management Program Support Project, the SEED Project (Kafue National Park), the SEED Project (Tourism), and the Agriculture Development Support Project. The Bank engaged Synovate Zambia, a market research �rm, to carry out the third-party monitoring. The assessment team looked at project awareness, satisfaction, and implementation. The team also collected and sum- marized suggestions from stakeholders regarding the various projects. The Synovate team �nds that the projects suffer from delays in funding and levels of awareness for some projects range from high to limited. Complaints about sustainability, shifting project goals, and low levels of consultation emanate from stakeholders. Concerns arise over project imple- mentation. In one particular �nding from the team, some community members raise concerns ENHANCING TRANSPARENCY & ACCOUNTABILITY THROUGH CITIZEN FEEDBACK about pesticide spraying to control malaria; they say that the parties responsible for spraying households dilute sprays in order to keep the surplus for resale and that this phenomenon leads to some community members’ resistance to having their homes sprayed. E-ISR+ experience in Sierra Leone E-ISR+ in Sierra Leone covers third-party monitoring of aspects of two projects: the Rural and Private Sector Development Program and the Decentralized Service Delivery Program. The Local Government Development Grant Program was also covered, although no site visits were con- ducted. In this case, the Bank contracted with an individual consultant, who worked with civil society groups, to produce the report. Among the �ndings, the consultant reports direct bene�ciaries and community leaders are not 12 always well-informed about project progress, but that there are signi�cant bene�cial impacts of the feeder road project, including signi�cant social impacts. The report also �nds that delayed disbursement of funds has led to overall project delays. E-ISR+ experience in Ghana E-ISR+ in Ghana includes third-party monitoring of six projects: the Urban Water Project, the Community-Based Rural Development Project, the Natural Resource Environmental Gover- nance Project, the National Health Insurance Authority Project, the Land Administration Project, and Oil & Gas Technical Assistance. The Bank contracted with the Ghana Anti-Corruption Coali- tion to coordinate and supervise civil society organizations in the M&E process. Feedback was solicited through focus group discussions and other inputs. In its �ndings, the assessment team reports project bene�ciaries and civil society organizations have limited knowledge of project objectives, but in some cases CSOs knowledge, when appli- cable, is not being adequately tapped in projects. Government of�cials �nd it dif�cult to accept the role of civil society organizations in evaluations. In addition, the public is not always well informed about projects set up to streamline service delivery. Ultimately, continuous assessment and evaluation of projects will enable public institutions to be more open and transparent in sharing information on projects they are implementing. E-ISR+ experience in Burkina Faso E-ISR+ in Burkina Faso encompasses four projects: the Urban Water and Sanitation Project, the Transport Sector Project, the Agriculture Productivity and Food Security Project, and the Agri- culture Diversi�cation and Market Development Project. In addition, the third-party monitoring initiative was scaled up to cover 5 recipient-executed trust funds in Burkina Faso. Selected third parties were also requested to prepare questionnaires and conduct collect feedback prior to supervision missions. Two distinct approaches were used in the assessment: a private �rm coordinated the effort for three projects, and an umbrella civil society organization in the water sector coordinated the initiative for the fourth project. To date, the use of the �rm conducting the third-party monitor- ing exercise has enabled the initiative to be managed by a competent, capable, and neutral interlocutor that has in turn enhanced the legitimacy and credibility of the results. By contrast, coordination of the initiative by the CSO umbrella organization has involved signi�cant delays in reporting due to capacity limitations. AFRICA OPERATIONS SERVICES SERIES The initial �ndings of the �rm’s feedback reports show that bene�ciaries appreciate the opportu- nity to share their perceptions of project performance openly and frankly with an independent monitor, as long as their right to con�dentiality is observed. Conclusions of the �rst feedback reports have particularly helped project coordination units improve their communication strate- gies around speci�c topics to manage bene�ciaries’ expectations and better inform them on speci�c aspects of projects. The �ndings also report on less positive feedback. Project coordination units are reluctant to have projects externally scrutinized. This skepticism requires persistent sensitization to the advantages of integrating third-party monitoring into projects, for reasons not in the least to avoid project coordination staff from discrediting the third-party feedback. Other challenges include making timely provision of information by the project coordination units to the third-party monitors allowing reasonable time for �eld visits and production of quality deliverables, followed by the 13 high transaction costs for staff time in facilitating exchanges between the project coordination unit and select third parties to provide for succinct and pertinent feedback reports. E-ISR+ experience in DRC and Congo-Brazzaville In DRC, a Demand for Good Governance concept note includes E-ISR+ in its development. The team identi�es civil society organizations with the capacity to monitor aspects of projects, including budget and procurement monitoring. A country-wide approach is being developed that will obtain project feedback at the province level and build upon ongoing multi-stakeholder consultations. In Congo-Brazzaville, the Congolese Government has asked the Bank to support a demand-side approach to good governance utilized throughout the entire portfolio of the Government. Exter- nal feedback has been incorporated in the Water, Electricity and Urban Development Project for FY12; this was collected during consultations with a wide range of stakeholders, including local development committees, community leaders, and civil society. Other projects, including agriculture, HIV-AIDS and the governance project, have already incorporated feedback from bene�ciaries, academics, and others. AFRICA OPERATIONS SERVICES SERIES V. Lessons learned from E-ISR+ initial phase This chapter is based on country reports, E-ISR+ documentation; and various inputs from Bank staff who have been involved in the rollout of the pilot E-ISR phase, the subsequent E-ISR+ exercise, or both. Since E-ISR+ is still in its early trial phase, this is not meant to be a comprehensive 15 list of lessons learned but rather part of an ongoing compilation and re�nement process. Input into the process will continue as E-ISR+ is rolled out to more countries in Africa and beyond. Aspects of E-ISR+ that worked well Civil society organizations build capacity in �nding and analyzing information. For those proj- ects that did contract with civil society organizations to carry out monitoring, civil society capac- ity did increase, particularly with respect to ability to access, analyze, and act upon information regarding both World Bank and government projects. Oftentimes, even when the Bank and/or government attempt to make information accessible, civil society organizations did not know where to go to �nd the information and, once found, how to comprehend it. Thus, working within the constraints of the E-ISR+ process helped, in some cases, to increase the ability of the civil society organization to carry out the process. This heightened ability to access information has positive spillover effects for accountability and good governance in general. In Burkina Faso, the initiative also gives legitimacy to the umbrella organization’s attempts to obtain information on project performance from the water authority. Civil society organizations build capacity in the technical aspects of monitoring and evalua- tion of projects. If E-ISR+ is to carry the potential bene�ts of building civil society capacity in the long term, it must also work to gradually increase the suitable pool of civil society players that have the technical capacity to carry out monitoring and evaluation of projects. Although not without some setbacks and/or challenges, working within the constraints of the formal E-ISR+ process has increased the formal M&E capacity of those civil society organizations to carry out the process. One Bank staff member notes that by the end of the second round of E-ISR+, the civil society group engaged by the Bank knew how to think through reporting mechanisms, reporting tools, data-gathering methods, and other aspects of monitoring and evaluation that they were not familiar with previously. Moreover, working with CSO coalitions builds broader capacity within the CSO sector and can mitigate some of the political risks of working with just one organization. Third-party feedback highlights ongoing project challenges that can, in some cases, resolve ongoing project challenges in current or subsequent phases. These problems may escape the notice of government implementers and/or Bank country management units; but if brought to the attention of the relevant people early enough, addressing challenges can deliver better results on the ground. Highlighting issues reported in the feedback �lls in missing information or provides an impor- tant counterweight to of�cial reports. Capable, knowledgeable, civil society organizations may ENHANCING TRANSPARENCY & ACCOUNTABILITY THROUGH CITIZEN FEEDBACK bring expertise about political economy issues surrounding speci�c projects. Different perspec- tives on obstacles and risks can be extremely valuable. E-ISR+, according to one Bank country of�ce staffer, can often capture more than “mission eyes� alone can see. Several country reports flagged important issues with respect to project bene�ciaries and weak links in the communica- tion process among stakeholders, the Bank, and the government. Others noted important aspects of the project that were not progressing as anticipated due to community mistrust or misunder- standing of the purpose or implementation of initiatives. More importantly, flagging issues at the implementation stage can sometimes lead to modi�cations in the projects. In one case in Zambia, a task team leader was actively engaged in the process, listened intently to the discussions and, with the government, devised a plan for addressing criti- cal issues. Other Bank staff note that reports have had the most impact when the team leader of the third-party monitoring group has one-to-one conversations with the project team leaders and/ 16 or government counterparts immediately after the conclusion of the monitoring exercise, leaving enough time to incorporate �ndings during the next round of decision-making in the project cycle. In Sierra Leone, some E-ISR+ recommendations were captured in the design and rollout of the second phase, or additional �nancing phase, of at least one project. The E-ISR+ report observa- tion states that community leaders were not always well informed about the pre-construction phase of feeder roads; this issue was seriously addressed in the subsequent phase of Bank support to the project. Third-party feedback leads to new or improved governance-related initiatives. E-ISR+ data has the most impact with respect to the demand-side governance work of the respective Bank coun- try of�ce. In at least one case, E-ISR+ was useful in helping to establish how and where to con- centrate efforts when piloting new demand-side governance initiatives. In Zambia, the E-ISR+ process directly influenced the conceptualization for a new irrigation support project. Third-party feedback leads to increased trust between civil society organizations and govern- ment in project implementation and a better country environment for good governance, trans- parency, and accountability. The E-ISR+ process supports, in a systemic way, the creation of more-lasting platforms for civil society and government to interact in an ongoing basis that may extend beyond the E-ISR+ process or the life of the project in question. In the case of Nigeria, which used a CSO network to carry out the monitoring process, the E-ISR+ process spearheaded the process of creating a platform through which civil society and govern- ment could interact in a tangible and regularized manner, thus setting the stage for potentially more lasting engagement. This platform entails equipping civil society with the right information and the right skills, and equipping government to understand the role of civil society in helping deliver services to the population. Building understanding and trust between these two parties through the E-ISR+ process is a �rst step toward a broader social accountability exercise. Moreover, civil society organizations were empowered by the interactions afforded by the E-ISR+ process. In addition to simply building technical capacity, E-ISR+ enabled these groups to feel as though they had the right to ask questions of the Bank and the government project implementa- tion units, to verify information, and in a general sense, to demand accountability for project shortcomings. When civil society feels empowered to demand answers of large institutions such as the Bank and the government, this can be a key component of building a broader environment to support greater transparency and accountability. Third-party feedback leads to heightened awareness of transparency and accountability surrounding Bank-funded projects. In general, E-ISR+ informs both the general public and AFRICA OPERATIONS SERVICES SERIES specialized audiences about existing transparency surrounding Bank projects and the potential for improved transparency. In Sierra Leone, E-ISR+ in combination with other social accountabil- ity initiatives helps raise awareness about Bank project-related transparency and accountability within the Bank, within government circles, and among civil society organizations in Sierra Leone. In other regions, Bank staff also feel that E-ISR+ helps to serve this function, although they caution that it would take multiple iterations and a speci�c focus on CSO capacity building in particular, for E-ISR+ to contribute to a heightened transparency and accountability environment in a speci�c country. Aspects of E-ISR+ that presented challenges Ambiguity regarding the purpose of E-ISR+ has created some confusion about contracting with 17 implementers. The teams that carry out the E-ISR+ project monitoring exercise are generally well prepared, but a lack of clarity in some cases regarding the ultimate purpose of E-ISR+ leads to Bankwide differences in implementation. This is particularly evident with respect to deciding whether or not to contract out the monitoring to individuals, CSO groups, or professional �rms. It is not clear if E-ISR+ is pure monitoring and evaluation; monitoring and evaluation with some stakeholder engagement; strengthening bene�ciary capacity; a social accountability mechanism; or some combination of all of these. Without �rm guidance about how the process should be undertaken, different countries have adopted different methods, with different results. Concerns about conflict of interest derive from the ambiguity issue. A common concern arising from the ambiguity is the issue of conflict of interest. In any situation where the Bank contracts an outside �rm to evaluate Bank-�nanced projects, there is a potential conflict of interest. In some situations, this potential is heightened in the case of contracting with civil society organizations or other non-professional groups. Civil society organizations can act as implementing agents but also as agents of oversight, and the two roles may come into conflict. For instance, some stakeholders and/or bene�ciaries could have ulterior motives or �nancial interests in influencing projects in one direction or another. If they are to be included in the M&E team, it could open the Bank to accusations of improper conduct. On the other hand, for E-ISR+ to serve as an exercise for stakeholder engagement, social accountability, or capacity building, the process by necessity should engage not with professional private sector �rms but with actual stakeholders. And these stakeholders are, in most cases, the civil society organizations and bene�ciaries. As another example of potential conflict of interest, in many cases, the E-ISR+ process relies heavily on the characteristics of individual project task team leaders and Bank staff who are required to drive the process; de�ne questions, locations, and stakeholders for evaluation; and decide on whether to adopt the recommendations presented by the group. Because of this strong guidance role (which is not necessarily a bad thing), the outcome of the EISR+ process can be potentially influenced by the leadership’s possible “unspoken� partiality to collect mainly “good examples� and hear from the least-critical stakeholders. On the other hand, if the right civil soci- ety organizations—the ones with the capacity to be objective and uninfluenced—are selected to carry out the exercise, there is the good chance they could help “balance out� the interests of task team leaders who may want to “cherry-pick� only successful projects for E-ISR+. Conflict of interest issues could become further complicated once external �nancing for E-ISR+ ends and it is required to become self-�nancing. In that case, its independence as a monitoring mechanism could become further compromised. However, with this concern and the others raised in this section, it is important to keep in mind that the objectives of traditional monitor- ing and evaluation and participatory monitoring and evaluation can be very different (as noted in Chapters 2 and 3 of this note). If future E-ISR+ is intended to perform the traditional function ENHANCING TRANSPARENCY & ACCOUNTABILITY THROUGH CITIZEN FEEDBACK of objective monitoring and evaluation, without any stakeholder outreach or capacity-building aspect, this points in one direction; if it is intended to boost stakeholder participation and capac- ity, this points in another. To help resolve the potential conflict of interest issues once and for all, the conceptual mission of E-ISR+ must be clari�ed and agreed upon both within the Bank and with its partners and stakeholders. Dif�culty identifying civil service organizations with capacity to conduct the E-ISR+ exercise reduces the pool of potential monitors. Country management units employ private market research �rms and individual professional consultants in most cases when the choice is limited in their favor. In such cases, there are insuf�cient civil service organizations with suf�cient ana- lytical, writing, and M&E skills to undertake the task. While some national-level civil society organizations do possess that kind of capacity, it is more dif�cult to �nd the necessary skills in state- or local-level organizations to monitor local-level projects. In a case in Nigeria, the CSO 18 network has substantial experience interacting with the Bank and had a large local network, which it could engage for the E-ISR+ exercise. Even with this organizational spread, coordination between the network’s national and local levels during E-ISR+ was not easy. In any kind of CSO capacity-building exercise of this sort, there is by necessity a chicken-and- egg type of dilemma. While skills are lacking, project managers are wary of entrusting the civil society organization with responsibility; but at the same time, the organization must be entrusted with some responsibility in order to build professionalism and skills. The E-ISR+ process has the potential to help build CSO capacity, but it must address these challenges head-on. Contracting with civil society organizations can be time-consuming and costly, and civil soci- ety organizations may lack suf�cient internal �duciary or governance mechanisms. Often local organizations do not have internal contracting mechanisms (or governance and manage- ment capacity) to contract directly with the World Bank. Some form of internal �duciary or gov- ernance capacity building for the civil society organizations may be necessary before they can deal with entities such as the World Bank; however, this capacity building can itself be costly and time-consuming. Without dedicated funding, this aspect is dif�cult to achieve. Add to this mix is the fact that the pool of available civil society organizations is oftentimes low. For this reason, the easier route for Bank country management units might be to contract directly with individual expert consultants or private �rms during the initial phase of E-ISR+. On the other hand, if E-ISR+ is intended to help build capacity over the long term, the challenge centered on selection of civil society organizations must be addressed. In such cases to boost selection, the Bank should perform pre-contractual due diligence on civil society organizations. It may be helpful for the Bank to support country management units more actively and directly in the procurement process. Coordinating and getting buy-in from all relevant parties (national and local civil society orga- nizations, government project implement units, Bank task team leaders, and country manage- ment units) can be dif�cult. Even in cases where civil society organizations are well-managed and well-governed and capable of conducting research and analysis, coordinating among the various parties can prove challenging, particularly if this requires coordination between national, state, and local levels. In Nigeria, for instance, a CSO network was selected to implement E-ISR+. This network is also mandated to interface with other CSOs on behalf of the World Bank. In this case, the national CSO network selected the local-level CSOs that participated in E-ISR+. The latter groups collected data on the ground with the support of the national-level organizations. Because the CSO network was already in place and had already acted as a trusted partner with the Bank, the buy-in from the host of players was made easier, although some coordination chal- lenges remain. AFRICA OPERATIONS SERVICES SERIES Getting Bank-side buy-in can also present its own challenges. In some cases, lack of explicit buy- in or support from the task team leader or country management unit impedes progress in E-ISR+. Some team leaders or management units feel that their existing social accountability initiatives are suf�cient and thus do not want to invest additional time or resources in E-ISR+. This issue may present even more of a challenge when there is no external source of funding or support for the E-ISR+ process. Some projects are better suited to E-ISR+ than others. The E-ISR+ experience shows that proj- ects with direct impact (i.e., health service delivery) seem to elicit better results and feedback than projects with more indirect impact (i.e., public sector reform). Bene�ciaries generally seem less aware of projects with indirect impacts, perhaps because it is more dif�cult to identify direct “bene�ciaries� of projects with indirect impact, or with impacts that are dif�cult to properly evaluate from a layman’s perspective. Because of this, projects with indirect impact tend to 19 receive worse feedback as a result. This could eventually prove alienating to the team leaders of projects that are more dif�cult to monitor and evaluate from a bene�ciary perspective, and could back�re the support for E-ISR+. Because some projects seem better suited to E-ISR+, the process may need to be tailored dif- ferently for different projects, with special care put into thinking through the M&E process for projects with more indirect or diffuse impacts. AFRICA OPERATIONS SERVICES SERIES VI. Emerging good practices for E-ISR+ Since E-ISR+ is still in the early stages, and evaluations of the process are still forthcoming, the emerging practices are in progress and still adaptable. The suggested good practices in this chapter are based on country reports, E-ISR+ documentation, and other inputs from Bank staff 21 who were involved in the rollout of the pilot E-ISR phase, the subsequent E-ISR+ exercise, and other aspects of the process. a. Consider broader participation, open development, and social accountability outcomes when designing and setting up E-ISR+ processes. While some ambiguity still exists about the exact goals of E-ISR+ and some variance persists about the form E-ISR+ takes from country to country, it is absolute that E-ISR+ is not just a supplementary M&E exercise but is part of a broader transparency and accountability agenda at the World Bank. E-ISR+ has the potential to have larger impacts on CSO capacity and partici- pation, as well as transparency and accountability surrounding Bank and government projects more broadly within individual countries. For this to happen, these goals must be envisioned and articulated during the design phase of the E-ISR+ process. The decision, for instance, about whether to hire a professional market survey �rm, an individual, a civil service organization, or a CSO consortium to undertake the work can have consequences for the broader E-ISR+ impact on the transparency and accountability agenda. In Burkina, when recipient-executed trust fund coordinators were �rst introduced to E-ISR+, some coordinators were cautious and asked many questions about the selection of the third-party monitor, its pro�le, and how it intended to work. Once the World Bank team had provided the requested answers and information, the coordi- nators felt more comfortable with the approach and showed more assurance in the third-party monitor. Such considerations about the quali�cations and selection of the E-ISR+ monitor should therefore be anticipated and planned for during implementation. b. Anticipate CSO capacity issues at the outset and plan for them. If civil society organizations are part of the process and in need of capacity building to do the job, consideration of the types of capacity-building measures should be a part of the front-end process. Necessary steps should be taken to ensure that any contracted CSO groups have the necessary internal accountability, �duciary, and governance structures, as well as the necessary technical capacity to carry out the process. These types of activities may also �t within other social accountability measures already being implemented by country management units. c. Get buy-in from relevant government project implementation units and civil society orga- nizations from the beginning. E-ISR+ functions best when all parties involved are on the same page. Early buy-in and participa- tion should come not just from the Bank, but also from civil society organizations and govern- ment project implementation units. Building a platform of trust among project partners and civil society groups will help the E-ISR+ process run more ef�ciently and smoothly. It can also have ENHANCING TRANSPARENCY & ACCOUNTABILITY THROUGH CITIZEN FEEDBACK important after-effects, particularly if it establishes channels for dialogue between government and civil society that outlasts individual project cycles. In Burkina, for instance, trust was successfully established between project implementation units and third-party monitors, and helped to get buy-in from more skeptical project implementation units. The third-party report was showcased and used as a template for other third-party-moni- tored projects. d. On the Bank side, encourage active TTL and CMU involvement while still maintaining enough distance so results are credible. Bank staff who have been involved with implementing E-ISR+ note that the ultimate success of the process depends heavily on the characteristics of an individual task team leader, and the 22 extent to which that team leader supports the process and buys into the concept of downward accountability. In some cases, an enthusiastic task team leader can drive the success of the process. In other cases, an unmotivated TTL, who is skeptical about the value of engaging with project bene�ciaries, may lead to a super�cial process that ultimately leads to breakdown of trust between the community and the project. Task team leaders are in a position to exert tremendous influence over the entire E-ISR+ process. Because the team leaders de�ne the questions to be asked and identi�es project location and the bene�ciaries to be surveyed, he or she can influence the outcome of the evaluation in a number of ways, both positively and negatively. For instance, they have the ability to skew the process so that only “good� projects are selected for evaluation by E-ISR+. Even if a TTL does not explicitly attempt to do this, there may be an underlying incentive for the process to identify and focus on good examples, thus generating additional positive feedback for the Bank. On the other hand, a task team leader who does not engage with the process may compromise the quality of the evaluation process, by failing to de�ne relevant questions and not identifying suitable projects or bene�ciaries to target. Moreover, their buy-in is necessary if the recommen- dations made by the evaluation team are ever to be taken up in practice. On the Bank side, CMU and sector manager buy-in is also important to the success of E-ISR+. e. Ensure dedicated funds and other types of support for the process. In its initial phases, external funding made it possible for country of�ces to pilot E-ISR+. In the future, dedicated funding sources will have to be identi�ed and built in to the design of the process at the country level. This goes hand in hand with ensuring committed support from task team leaders and country management units, since task team leaders have an important role from a logistical and �nancial perspective. Time commitments must also be built into the process. It may take more time to vet and hire civil society organizations to carry out the E-ISR+ process than to employ private sector �rms that already have a working relationship with the Bank. Bank staff who have implemented the initial phase of E-ISR+ note that the Vice Presidential Unit should provide ample time and technical support in procurement issues, particularly with respect to civil society organizations. f. Build in incentives for follow-up. Although some countries that participated in the initial phases of E-ISR+ see tangible adjustments to projects as a result of the feedback generated, not all did. Some note that there is little incen- tive built into the E-ISR+ process for project teams to act on the feedback generated. In some AFRICA OPERATIONS SERVICES SERIES cases, the same issues have been raised consistently in the same projects with little follow-up. To some extent, recommendation follow-up is dependent on personal attitude (i.e., TTL atti- tude) and initiative rather than on structural incentives for following up on E-ISR+ feedback. This may pose an existential question for the E-ISR+ process: is it designed merely as another feedback mechanism, or should it incorporate tangible avenues for that feedback to translate into improved project results? Those projects that did see follow-up of recommendations are careful to ensure regular coor- dination and communication among the evaluation team, project implementation units, Bank task team leaders and country management units. Planning for speci�c meetings between these counterparts at the end of the E-ISR+ process can ensure follow-up of recommendations. In sum- mary, the presence of some formal component to recognize feedback and discuss its implica- tions can help E-ISR+ to have a bene�cial impact on project results. 23 AFRICA OPERATIONS SERVICES SERIES References Estrella, Marisol, and John Gaventa. 1997. “Who Counts Reality? Participatory Monitoring and Evaluation: A Literature Review.� IDS Working Paper 70, 1997. http://www.ids.ac.uk/�les/ Wp70.pdf IFC (International Finance Corporation). 2007. Stakeholder Engagement: A Good Practice Hand- book for Companies Doing Business in Emerging Markets. http://www.odi.org.uk/resources/ docs/2121.pdf. Malena, Carmen, with Reiner Forster, and Janmejay Singh. 2004. Social Accountability: An Intro- 25 duction to the Concept and Emerging Practice. The Social Development Papers: Participa- tion and Civic Engagement. Paper No. 76. World Bank. Ringold, Dena, Alaka Holla, Margarett Koziol, and Santhosh Srinivasan. 2012. Citizens and Ser- vice Delivery: Assessing the Use of Social Accountability Approaches in Human Develop- ment. Washington, D.C.: World Bank. http://www.odi.org.uk/events/docs/4871.pdf. World Bank. 2006. World Bank Social Accountability Sourcebook. http://www.worldbank.org/ socialaccountability_sourcebook/.