80452 DEALING WITH GOVERNANCE AND CORRUPTION RISKS IN PROJECT LENDING http://dfgg http://gacinprojects HOW-TO NOTES Participatory and Third Party Monitoring in World Bank–Financed Projects: What Can Non-state Actors Do? This note was prepared by Warren A. Van Wicklin III and Asli Gurkan of the World Bank’s Social Development Department (SDV). It is part of the efforts by SDV’s Social Accountability team and the Operational Policies and Country Services (OPCS) Governance and Anti-Corruption (GAC)-in-Projects team to provide guidance on ways to improve governance and accountability in World Bank operations. Zeynep Darendeliler (SDV) conducted the stocktaking review of monitoring by non-state actors through interviews with 35 Bank staff and wrote the report based on that research. Asli Gurkan managed the preparation of this note and the stocktaking review. Hélène Grandvoinnet (SDV) provided overall guidance and comments. Saki Kumagai (SDV) provided project examples of third party monitoring using information and communication technologies (ICTs). The authors are grateful to peer reviewers Ditte Fallesen (Afghanistan CMU), Balakrishna Menon (MNSSO), Caroline Sage (AFTCS), and Diane Zovighian (AFTCS). Ditte Fallesen also provided additional suggestions on monitoring in fragile and conflict-affected states. The authors also thank Marie Brown (OPCS), Stephan Eggli (OPCS), Richard Holloway (SASDS), Sarah Keener (LCSSO), Luiza Nora (SASDS), Meg McDermott, Norma Garza, Lena Krylova, Sara Danish, and Marcela Rozo (WBIOG and WBISG) for providing insights and comments. Participatory and Third Party Monitoring in World Bank–Financed Projects  1 Introduction Many World Bank task teams wonder how they can better understand what is actually hap- pening in their projects. Relying only on the project implementing or monitoring unit may be insufficient and leave task teams searching for more information. This note describes ways to complement regular monitoring by the government or project implementing units with monitoring by communities, civil society organizations (CSOs), research institutes, consult- ing firms, and others broadly defined in this note as “non-state actors� (NSAs).1 Monitoring conducted by NSAs, referred to in this note as, “non-state monitoring� (NSM), could sig- nificantly improve task team knowledge about project implementation, impacts, targeting, and problems; gain the support of project beneficiaries and other stakeholders; give credibility to project findings; and ultimately improve project development effectiveness. NSM can be par- ticularly useful in conditions of limited access, such as fragile and conflict areas and projects with numerous sites to monitor. This note offers a process-oriented, step-by-step guide to designing and implementing NSM of Bank-financed projects. It provides guidance on: (i) identifying project monitoring needs that would benefit from monitoring by NSAs; (ii) assessing the country and local context; (iii) select- ing the monitoring methodology or approach that fits the context; (iv) designing the implemen- tation modalities for NSM; (v) implementing monitoring by NSAs; (vi) monitoring and assessing the effectiveness; and (vii) institutionalizing, scaling up, and sustaining NSM. Targeted mainly to Bank project task teams, this note is not intended to provide quick fixes or ready-made solutions. It is aimed more at laying out a menu of options, a wide range of issues to consider in designing and implementing monitoring approaches by NSAs, and in selecting potential organizations that project teams can work with.2 The next section describes the two main forms of non-state monitoring (NSM): third party monitoring (TPM) and participatory monitoring. The third section elaborates a wide variety of uses of NSM with project examples. Then the core guidance is presented in a series of sections on a step-by-step process approach to the design and implementation of NSM. A final section provides additional tips on implementation. Annex 1 contains a table suggesting range of options that can be chosen based on monitoring objectives and the project context. Annex 2 provides a checklist of design and implementation steps. Annex 3 provides a brief description of six common tools used by NSAs in project monitoring, including their objectives, benefits, uses, limits, challenges, and implementation tips. For this note, non-state monitoring (NSM) is defined as a process where parties other What Is Non-state than the state and donors track the implementation of development projects or programs and Monitoring? obtain beneficiary feedback to increase accountability to the beneficiaries.3 NSAs, according to this definition, include beneficiaries, communities, CSOs, think tanks, research institutions, academia, media, for-profit firms, labor and business associations, voluntary associations such as school boards, and other groups. NSM can be seen as an umbrella concept to represent a continuum from participatory monitoring to third party monitoring. NSM often combines ele- ments of both because they complement each other.4 1. The note uses “non-state actors� instead of CSOs to include private sector or other for-profit entities. 2. The Social Accountability Community of Practice and the SDV Social Accountability Team (contacts: Hélène Grandvoinnet and Asli Gurkan) are good starting points for seeking additional information. The Social Account- ability database (FURL: “Saccdb�) includes a number of practical documents including good practices, manuals and terms of references. 3. “Non-state monitoring� is the term this note adopted because there was no widely accepted existing term to describe the concept. The more established term is third party monitoring. However, third party monitoring by defini- tion denotes an arm’s length relationship with project management structures. In practice, however, communities and project beneficiaries can themselves play an active role as monitors and therefore are not at arm’s length in their assessment of project implementation or performance. NSM is meant to be an umbrella concept that covers a variety of complementary and blended methods ranging from participatory monitoring to third party monitoring. 4. The tools and approaches used for participatory and third party monitoring are often the same. The difference between the two is ultimately defined by “who� does the monitoring. 2  How-to Notes Participatory monitoring refers to the active participation of project beneficiaries, project- affected people, communities, and other primary stakeholders in designing and implementing the monitoring. This definition goes beyond having consultations with primary stakeholders on predefined indicators, or asking them to provide information or feedback. It is built around agreeing on expected results, defining jointly with project beneficiaries about how to track progress, collecting required data, undertaking analysis, and developing practical action plans to resolve identified problems. Participatory monitoring is generally used to increase the voice and ownership of primary stakeholders, to improve the relationship between the state and citizens, and to hold the state and service providers accountable. It can be further differentiated between (i) monitoring by local committees that include community representatives, and (ii) monitoring that directly involves beneficiaries through face-to-face meetings or information communication tech- nologies (ICTs). The involvement of local stakeholders is both useful for observing and identify- ing the root causes of problems through their familiarity with local issues, and for increasing the chances of sustainability of activities beyond the duration of the project. On the other hand, participatory monitoring (particularly the type that relies on local committees) runs the risk of elite capture, and may not be representative or inclusive. Third party monitoring (TPM) is defined as monitoring by parties that are external to the project or program’s direct beneficiary chain or management structure to assess whether intended outputs, outcomes, and impacts have been achieved by the project. TPM is mainly used to provide an independent perspective on project or government performance. It can be conducted by CSOs, think tanks, academic institutions, media, or private firms. These organi- zations generally have greater skills for monitoring than community representatives. However, there are large variations in skills, for example, between a firm that specializes in survey tech- niques and a grassroots CSO that specializes in social mobilization, advocacy, and facilitation. TPM usually involves project beneficiaries and at a minimum solicits their views in order to gather evidence and triangulate information. Yet, the findings and conclusions of third parties may not be fully aligned with the views of project beneficiaries or communities, because they are ulti- mately meant to be the independent judgment of the organization conducting the monitoring. The continuum of monitoring approaches used by non-state actors (NSAs) varies on dimen- sions other than from participatory to third party monitoring. They can focus on monitoring processes during project implementation, such as how the contractors in a road project are disseminating information or interacting with the local communities on issues such as creating local employment. Or NSAs can focus on monitoring outputs and outcomes, such as miles of road constructed and usage. NSM can emphasize qualitative or quantitative methods. Participatory monitoring is generally associated with more qualitative methods and process concerns than third party monitoring. These two main types of NSM and their main purposes are illustrated in Figure 1. Ultimately, most Bank-funded projects that have worked with non- state actors for monitoring purposes use blended approaches. Participatory and third party methods should not be viewed as silos. Figure 1 Two Main Types of Monitoring Approaches Participatory Monitoring used by Non-state Actors Usually by primary stakeholders or community-level committees representing them Third Party Monitoring Usually by CSOs or firms Mainly used to give primary stakeholders voice in monitoring and Mainly used to provide an independent increase their ownership perspective on project performance and triangulate information/data Participatory and Third Party Monitoring in World Bank–Financed Projects  3 Why Is the World This section summarizes the reasons most commonly cited by World Bank staff for using non- Bank Group (WBG) state actors in project monitoring and provides project examples. There are three main reasons for using non-state monitoring: (i) increasing project effectiveness by obtaining more informa- Interested in tion on the quality and impact of project implementation; (ii) making government and service Monitoring by providers more accountable to beneficiaries through greater empowerment and ownership; Non-state Actors? and (iii) practical reasons relating to the limitations of conventional monitoring approaches. Third party monitoring arrangements have often been used to fight fraud, corruption, and con- flicts of interest that transcend all these rationales. These three main reasons are discussed in order, with examples of NSM uses for each. (i) Increase project effectiveness by obtaining more information and verification NSM is used to get a more complete sense of how the project is performing compared to the information that standard monitoring approaches provide. Standard monitoring frameworks tend to quantitatively measure results and impacts, but they do not always detect the more subtle issues in project implementation such as how people are affected by the project, how contractors and government staff interact with local communities, or how communication and awareness campaigns reach (or fail to reach) targeted groups. This is especially important for results-based financing where disbursement is made conditional on project performance, such as the Program for Results (P4R).5 Both third party monitoring and participatory monitoring provide additional perspectives and a more complete picture of project performance, impacts, and results. For example, a number of pilot countries for the External Implementation Status Reports Plus (EISR+) Initiative in the Africa Region involved CSOs in monitoring Bank-financed projects. Improve resource use and targeting though local knowledge. One way NSM works is by tapping into community knowledge and oversight. This has proven especially useful for verification of intended resource use, and for monitoring accurate targeting of project benefits (such as conditional cash transfers, scholarships, and stipends) to groups and individuals. Similarly, communities are well positioned to enforce regulations at the local level. They can observe violations of restrictions on local natural resource management to prevent unsustain- able use or extraction. Box 1 Getting Feedback: External The Africa Region started the EISR+ Initiative in 2010 to disclose current project information to Implementation Status external stakeholders, to obtain feedback from beneficiaries on project progress and results, and to make timely adjustments based on that feedback. In Zambia, a market research firm Reports Plus Initiative in the assessed project awareness, satisfaction, and implementation, and collected suggestions from Africa Region stakeholders. In Ghana, the Anti-Corruption Coalition coordinated and supervised CSOs con- ducting monitoring and evaluation. In Nigeria, the Civil Society Consultative Group assessed the level of community participation in decision making, beneficiary satisfaction with project implementation and service delivery, and project outputs and outcomes. In Burkina Faso, a private firm and an umbrella CSO prepared questionnaires and collected feedback prior to Bank supervision missions. The EISR+ initiative has revealed some important lessons: Positive outcomes included: (i) increased knowledge and capacity of CSOs to monitor Bank projects, (ii) increased transparency and accountability around Bank projects, and (iii) increased trust between CSOs and government. Challenges included: (i) ambiguity of objectives and conflict of interest, (ii) low CSO capacity, (iii) difficulties in obtaining buy-in from project implementation units, and (iv) difficulties with procurement and financial management arrangements. Source: World Bank 2013. 5. For example, in the Morocco National Initiative for Human Development (INDH) 2 Program, a P4R program, the World Bank plans to hire a competitively selected firm to conduct independent verification for six of the nine disbursement-linked indicators. 4  How-to Notes Box 2 Using School Committees The Education Sector Support Scale-Up Action Program in Cambodia seeks to increase enroll- to Monitor Scholarship ment and retention in primary schools. As part of the program, students from families in need of assistance receive scholarships in order to encourage school attendance. The project uses Targeting and Recipients in both School Committees and a private firm for monitoring purposes. Local School Committees Cambodia are chosen at schools according to project guidelines. Students fill out eligibility questionnaires, which are read out loud in class and verified by students and teachers. These are used to gen- erate a list of potential scholarship applicants and posted in public areas around the schools. With the help of the School Committees, the selection is finalized. Complaints and attendance data are then collected as part of the impact evaluation by the private firm. As a result, local knowledge informs the targeting of beneficiaries. Provide an independent perspective on safeguard implementation. The Bank’s envi- ronmental and resettlement policies recommend third party monitoring of the implementation of the environmental management plan and the resettlement action plan. Because of the sen- sitivity of safeguard policies and reputational and developmental risks involved in weak compli- ance, third party monitoring may be helpful in independently verifying that adverse impacts of the project are properly mitigated and positive impacts are accurately captured. Because safe- guards often include fairly technical issues, safeguard monitoring is usually done by specialized institutes and consultants (see Box 3). For example, in China, universities and engineering design institutes are often used for monitoring of resettlement action plans. Box 3 Using a Local Consulting Under the Turkey Railways Restructuring Project, railway modernization and improvement in Firm to Monitor the Social infrastructure activities triggered the Bank’s involuntary resettlement policy. To independently monitor Resettlement Action Plan (RAP) implementation, a local consulting firm was contract- Impacts of a Railways ed. The firm visited project sites and held consultations with the project-affected people. Their Project in Turkey report found that some project-affected persons were satisfied with the project for creating job opportunities for young people in the project area. However, other project-affected persons reported not being sufficiently aware of the project implementation details. Nor were they satisfied with the partial expropriation of their lands, as the remaining parcels they owned were no longer economically viable for commercial or agricultural use. The external report also re- vealed certain hazards occurring during project implementation, such as compromised road safety for villagers and school children near construction sites. The firm discussed its findings with the project implementers during a World Bank supervision mission in the presence of Bank staff. The parties then developed a timetable to address the issues identified in the third party report. (ii) Empower beneficiaries and increase their ownership of development projects and programs Standard monitoring, with its main focus on project outputs and objectives, may not capture beneficiary feedback (see Box 4), incorporate their perspectives on project implementa- tion, or keep a pulse on community satisfaction. Non-state monitoring, especially participatory monitoring, can focus on how beneficiaries or other people are affected by or relate to the project. When people have greater voice and can make inputs into project decision making, Box 4 Direct Feedback from The main objective of the Karnataka Beneficiary Verification System (BVS) is to use beneficiary Beneficiaries: The India feedback to verify that services are reaching their intended recipients (pregnant women and children) in a timely and reliable manner. The BVS directly contributes to beneficiary engage- Karnataka Health Systems ment and feedback, strengthens client capacity to monitor and manage for results, and en- Development Project ables collection of data on outputs and outcomes, which helps health policy makers and providers monitor effectiveness. Reliable data from the BVS also helps government and Bank task teams plan their supervision missions better and focus scarce resources in areas that need attention. Feedback from health workers at the grassroots level on the challenges of service delivery (for example, shortage of supplies, lack of interest in local communities, and lack of adequate training) is building managerial capacity to improve program implementation based on evidence. Participatory and Third Party Monitoring in World Bank–Financed Projects  5 they have a greater sense of ownership of the project. Beyond the instrumental value of bet- ter project outcomes, NSM has intrinsic value through the empowerment of beneficiaries. Facilitating more engagement of non-state actors can also increase trust between them and government, and increase social capital. (iii) Complement project monitoring in areas with limited security, accessibility, and capacity NSM is often used where security concerns limit access by the government, the Bank, or other donors and agencies to monitor project implementation (see Box 5). Local non-state actors may be politically acceptable, know the local security situation, and be able to conduct moni- toring that others cannot due to their proximity to the project sites. Even non-state actors may have limited access that hinders the quality of their work, and it can be difficult to verify or probe the findings and results of NSM in such contexts. This is particularly the case in fragile and conflict- affected states (FCS), which is discussed under Design Steps. Box 5 Using Civil Society Under the Philippines Mindanao Reconstruction and Development Project, communities in Organizations to Monitor conflict-affected areas, internally displaced persons, and rebel returnees decide on, plan, and implement subprojects in order to restore their access to basic services. The subprojects are scat- Projects in Conflict-Affected tered over remote and conflicted areas where World Bank staff cannot travel. Therefore, the project Provinces of the Philippines is working with two CSOs to monitor project components that cannot otherwise be monitored by the Bank. The CSOs’ local knowledge (including local languages) makes monitoring possible. Complement monitoring in projects that have numerous, decentralized, or dispersed subprojects. Many Bank-financed projects are implemented in so many communities that it is impractical and perhaps impossible for the government to monitor the project in every location (see Box 6). This can be particularly the case of community-driven development projects, if communities are implementing the projects with relatively little government presence. Similarly, other projects are implemented over too large an area for centralized monitoring to be feasible or cost effective. Box 6 Using School Boards to The Support for Basic Education Sector Reform Project in the Philippines aimed to improve qual- Monitor School Construction ity of education across 44,000 schools nationally through school construction/repairs and de- livery of textbooks. The monitoring of the procurement and delivery of textbooks at the national and Textbook Delivery in level would have exceeded funding constraints due to the centralized nature of the Ministry of the Philippines Education. A national umbrella CSO was hired to lead implementation, and established School Governing Boards at each school composed of parents, teachers, students, and local NGOs. The date and content of delivery of textbooks was advertised in advance in one-page ads in local newspapers. The School Governing Boards then delegated members to monitor the delivery and count textbooks. The results were reported upwards from the local and municipal to the regional and national level, and made public. Monitoring school construction (especially quality) and textbook delivery would have been too expensive to do centrally, and community monitoring was considered to be more cost effective. Strengthen project monitoring when state agencies lack monitoring capacity. If the government or implementing agency cannot perform monitoring, then the Bank and other donors may turn to non-state actors to perform that function. NSM can be a useful way to build government’s monitoring capacity through a “learning by doing� approach. For example, in Afghanistan, the ministries involved in the third party monitoring program quickly caught on to this benefit. They were very enthusiastic about working with the international NGO to better understand the findings, get advice on how to address the issues identified, and to improve their own ability to perform similar work. Even in environments with large amounts of donor technical assistance, ministries can be eager to learn and take on additional work to build their capacity. 6  How-to Notes Potential Risks and Although NSM has many potential uses, it has risks and limitations: Limitations • As with standard monitoring, it can be difficult and costly to obtain valid and reliable data in assessing project performance. • Small samples and anecdotal findings might not be representative. • Because NSM is not done by project staff, it has the additional challenge of getting project management to take on board its findings. Project management may question the accuracy of findings that do not fit its view of the project, or may otherwise fail to act on them. • Poor-quality NSM can lead to inaccurate conclusions and if acted upon, potentially unwise decisions. • NSM can have risks other than reliability, such as elite capture (especially with participatory monitoring), creating false perceptions, or unfulfilled expectations. How Is Non-state NSM is used for a wide variety of tasks. Table 1 provides a few examples from a review of Monitoring Used in selected Bank-funded projects conducted for this note. NSM should not be limited to just these purposes and could be used innovatively if it appears appropriate and feasible. Bank-funded Projects? Table 1 Examples of Non-state Monitoring in World Bank–Funded Projects What is Monitored? How? By Whom? Countries/Projects Beneficiary Satisfaction Beneficiary surveys CSOs, • Bangladesh (Rural Transport Improvement Project) • General satisfaction with project (face-to-face and for-profit • Burkina Faso (E-ISR+ and Local Government implementation mobile phone surveys), consulting Support Project) focus group discussions, firms, • Jordan (Employer-Driven Skills Development) • Satisfaction with land reform and public hearings academia titling • Nepal (Rural Access Improvement and • Satisfaction with project Decentralization Project) communication, outreach • Romania (CESAR) • Satisfaction with treatment of • Uzbekistan (South Karakalpakstan Water Resources contractors Improvement Project) • Zambia (E-ISR+) Beneficiary Targeting Review of local Academia, • Afghanistan (HIV/AIDS Prevention) • Delivery of conditional cash transfers committee reports, for-profit • Cambodia (Education for All) community oversight, consulting • Kenya (Cash Transfer for Orphans and Vulnerable • Selection of scholarship recipients beneficiary surveys, firms, local Children) and absenteeism focus group discussions committees • Selection of cash-for-work youth • Philippines (Mindanao Reconstruction and laborers Development Project) • Employment skill programs • Sierra Leone (Youth Employment Support) • Targeting of HIV-infected groups • Yemen (Secondary Education Development and Girls Access Project) Quality of Service Provision Beneficiary surveys Academia, • Burkina Faso (multiple projects) • Quality of services in primary schools (mobile phones), focus local • Cambodia (Demand for Good Governance and health clinics group discussions, committees, Project) community scorecards, NGOs • Mongolia (Rural Education and Development) • Quality of service provision by district citizen report cards government • Philippines (Roads Watch) • Teaching quality of schools • Sierra Leone (E-ISR+) Delivery of Goods Community oversight, Local • Afghanistan Reconstruction Trust Fund • School textbook delivery including collecting committees, • Nepal (Rural Access Improvement and data via mobile phones international Decentralization Project) • Construction of schools and tablets, geo- NGOs • Philippines (Second Mindanao Rural Development • Construction and quality of roads tagging the data Project) • Contractor work in community-driven development projects • Philippines (Support for Basic Education Sector Reform) • Sierra Leone (Youth Employment Support) • Yemen (Secondary Education Development and Girls Access Project) (continued) Participatory and Third Party Monitoring in World Bank–Financed Projects  7 What is Monitored? How? By Whom? Countries/Projects Effectiveness of Grievance Redress Oversight of complaints Local • Bangladesh (Rural Transport Improvement Project) Measures handling (including committees • Sierra Leone (Youth Employment Support) • Cash-for-works job cards reports on bribes • Tanzania (Center for Economic Prosperity) submitted by mobile • Availability of work and promised • Yemen (Secondary Education Development and phones) compensation Girls Access Project) Resettlement Plan Implementation Beneficiary surveys Academia, • Sierra Leone (Bumbuna Hydroelectric • Resettlement compensation (face-to-face or via for-profit Environmental and Social Management Project) mobile phones), focus consulting • Turkey (Railway Restructuring Project) • Timeliness of payment group discussions firms • Vietnam (Priority Infrastructure Investment Project) • Effectiveness of procedures Budget Transparency and Allocation Public hearings, CSOs, • Bangladesh (Local Governance Support Project) • Decentralization of budgets simplified budget local • Burkina Faso (multiple projects) sheets, citizen budgets committees, • Sierra Leone (Integrated Public Financial • Municipal budget allocation used during meetings, professional Management and Reform Project) • Publication of national budget inputs into draft budget associations • Tanzania Secondary Education Development • Delivery of education capitation documents during grants budget preparation, Program social audits Procurement/Contracts Regular site visits, Firms, civil • Ghana (road sector) • Status of contract administration trainings, periodic society • Iraq (multiple projects) meetings with project coalitions, • Nigeria (power sector projects) • Compliance with WBG procurement monitoring units, local procedures • Philippines (procurement watch) separate fact sheets committees • Verification of whether staff and monthly summaries • Sierra Leone (Procurement Watch) mentioned in consultancy contracts of procurement are mobilized progress submitted to the Bank Natural Resources Management Community oversight, Local • Philippines (Second Mindanao Rural Development • Management of fishing grounds collecting data via committees Project) mobile phones and • Sierra Leone (West Africa Regional Fisheries • Sustainable resource use by tablets, geo-coding the Program) community data Source: Adapted and expanded from a similar table in Darendeliler (2012). More project examples can be found under the Social Accountability E-Guide (FURL: “SAEguide�) and the Social Accountability database (FURL: “SAccdb�). The Design of This section describes a step-by-step approach to the design of NSM. The four-step process Monitoring by is a guide to help project teams determine the applicability of NSM to their project needs and to incorporate it into their project. Figure 2 portrays the four main steps with some examples Non-state Actors and issues under each step. Figure 2 Steps in Designing 1. Identify the NSM 4. Design NSM 2. Assess the Context 3. Select NSM Non-state Monitoring Objective, Focus, and Target Audience for NSM Methods and Tools Implementation Details • Identify purpose • Political factors: • Context (conducive or • Data to be collected • Identify existing project government attitudes not) • Data collection methods accountability about and capacity for • Government/ • Organization to do NSM framework and gaps NSM community cooperation • Capacity building • Identify focus • Social factors: citizen required • Scale of NSM • Identify key questions and CSO capacity to • Champions present • Budgeting for NSM • Identify the target conduct NSM, cultural • Capacity needed • Funding source and audience norms • Complexity of NSM tools independence of NSM • Project-specific factors: • Cost and time required organization location, access, • Use of ICT security, number of sites 8  How-to Notes Step 1: Identify the main reasons for using non-state monitoring One needs to start with a reason for using NSM and then design the NSM system that will best meet that need, not vice versa. The earlier section on “value added� for using NSM and Table 2 covered the most common reasons. Not every project needs NSM, nor should every project use NSM. A good starting point would be to identify and analyze existing monitoring and accountability measures for the project. This helps to determine to what extent the existing framework is adequate and where there are gaps or the need for NSM to complement the exist- ing mechanisms in place. NSM often is used to close accountability loops. One early task is to decide what type of information is needed to help meet the monitoring objective. Listing the main questions that need to be answered is one way to identify the information required and where it can be obtained (see Box 7 for an example). Box 7 Key Questions Used for Third party monitoring reports by Nigerian CSOs shed some light not only on whether projects the Third Party Monitoring are being implemented and services delivered, but also on why things are working the way they are. Some questions they addressed included: Initiative in Nigeria • Why are there gaps in service delivery despite facilities and mechanisms being in place? • Why doesn’t information reach beneficiaries despite efforts to communicate about the project? • Why are certain groups not benefitting equally despite project team efforts to reach them? • Why are project inputs not used or misused? Is it due to beneficiaries’ lack of understanding? • Are there barriers to access? After defining the broad objective and existing measures, then the team needs to identify the focus of NSM. There tends to be two main areas of focus for NSM. • Project inputs and outputs that are the responsibility of government and/or the project implementing agency. The more technical focus areas—such as procurement, budgets, and expenditures—tend to use more third party monitoring approaches by more specialized and skilled organizations such as private consulting firms, research institutes, and academic institutions. • Project outcomes and impacts, and how various stakeholders experience or perceive those outcomes, especially project beneficiaries or affected people. The more qualitative focus areas—such as beneficiary satisfaction and perceptions—typically use more participatory approaches by organizations with deeper ties to, and the trust of, the beneficiaries. These include CSOs, community-based organizations, and local committees such as school boards and community development committees. Another key decision at this step is to identify the target audience for the information that NSM generates. This could include the project management unit, other organizations involved in implementing the project, national government, local government, project communities and beneficiaries, the Bank task team, and other stakeholders. The users of monitoring information are a critical factor in determining what kind of NSM is most likely to be useful and what format should be used for information sharing and reporting. This is when efforts to get buy-in from critical stakeholders should begin. If they have a role in designing NSM, they are more likely to support NSM implementation and use the findings. Step 2: Assess the project and country context for non-state monitoring The second step is to assess the project and country context to narrow down to activities that are feasible. The starting point can be to review previous experience with NSM in the country and project area, and by stakeholders that are likely to be involved with the proposed project. The context analysis6 could include: 6. To obtain more detailed advice on assessing the country and project context for social accountability initiatives, of which NSM is a subset, see World Bank (2011a). Participatory and Third Party Monitoring in World Bank–Financed Projects  9 • Political factors such as an existing legal framework on transparency and accountability, government willingness to be open to feedback, past track record, and capacity to work with NSAs • Social factors such as willingness of project beneficiaries to provide feedback and participate in discussions, citizen awareness and knowledge about the subject to be monitored, and cultural norms about engagement and accountability between state and citizens, as well as norms for interaction among different groups (male/female, youth/ elders, different ethnic, cultural groups living in the project area, and so forth) • Project-specific factors such as the location of the project area, especially in terms of access and security (for example, locations too remote or too unsafe due to security concerns for regular monitoring) Context analysis could be based on existing information or analysis, not a specially conducted assessment. Other Bank projects in the country may have already conducted social assess- ments with relevant information on different regions. For other country-level information, existing governance diagnostics, stakeholder analysis, poverty and social impact analyses, and other reports may be useful. Sometimes existing reports will not be sufficient. Then a frequent source of information are Bank country office staff, such as the NGO liaison officer, social development specialists, external affairs staff, CSOs, and other donors and development agencies operating in the country. They in turn may suggest experts that can provide a quick sense of the country, project, and sectoral context. What is essential for context analysis is visiting the project sites, meeting primary stakeholders to assess willingness and capacity constraints, and making per- sonal observations about their interactions and power dynamics. The most common constraints to NSM have been government reluctance about monitoring by non-state actors and lack of organizations with experience or skills in NSM. Sometimes government attitudes change, so this is not a fixed constraint, particularly if NSM can be linked to other reforms and ongoing initiatives. If the government is pursuing decentralization, broader sector governance, or anti-corruption reforms, this can create synergistic opportunities for NSM (see Box 8). Box 8 Linking Civil Society The Participatory Anti-corruption Initiative (known by its Spanish acronym IPAC) brought the gov- Monitoring to Anti- ernment, the private sector, and civil society together to elaborate a set of specific measures to fight corruption. Progress on the specific targets on multiple sectors was independently and Corruption Reforms: Case publicly rated by CSOs (red or green light), and compared to the government’s rating of its of the Dominican Republic own progress, similar to a scorecard process. The media have shown consistent interest in IPAC, publishing the implementation results and hosting key players on TV and radio programs. Since the inception of IPAC, over 220 related articles have appeared in the print and online newspapers. NSM in Fragile and Conflict-Affected States Fragile and conflict-affected states (FCS) are often different from “non-FCS� working environ- ments and they merit special attention. Implementation risks and capacity gaps are inherently heightened when working in FCS (see Box 9). It can be difficult or impossible to obtain or trian- gulate the data and ensure validity. Therefore, more flexibility is needed to make adjustments. The scale of NSM needed in a FCS context could be much larger than in other contexts. Depending on the NSM objectives, it is important to understand what the security situation allows, and how and whether this can be mitigated through NSM design. Governments in FCS often are hampered by low capacity for monitoring activities. Therefore it is important to consider early on how to ensure sufficient information flow and ownership. There are different 10  How-to Notes Box 9 Constraints to Third Party Given Afghanistan’s challenging political economy and security environment, the Bank was re- Monitoring in Fragile and quired to go through an extensive competitive bidding process to identify a third party monitor. A large, comprehensive program like the one in Afghanistan requires a range of different skills Conflict-Affected States: and high capacity in regard to program management and logistical operations. A detailed Afghanistan request for proposal helped identify the best agency to work with. On the one hand, interna- tional contractors and NGOs may have the necessary expertise but not the necessary detailed knowledge and understanding of the local context and what it requires to work in the field in Afghanistan. On the other hand, local contractors and NGOs suffer from capacity constraints, and may be politicized. In light of all these consideration and trade-offs, the Bank team chose an international NGO (International Relief and Development) with extensive experience in the country and established local networks. kinds of capacity constraints with respect to available contractors and CSOs that can perform monitoring. There is a need for more careful prior “market analysis� in FCS contexts to deter- mine who can conduct NSM in a trusted and reliable manner. FCS contexts are often characterized by a number of elements that may affect beneficiary participation in monitoring. Methods and approaches may need to be adapted. For example, questionnaires, focus groups, or interviews might need to be redesigned to suit the particular dynamics of these environments. The following aspects of FCS may affect beneficiary feed- back and participatory monitoring. • Community tensions: Conflict situations might be affected by intra- or intercommunity tensions or their dynamics. Gender, age, and ethnicity are important factors that should be analyzed and safe spaces provided to voice differences of opinion. • Community perceptions: The perceptions and role of the community in the broader conflict might affect their opinions of any external (project) intervention. • State-citizen relations: Fragile contexts can be affected by weak or even predatory state-citizen relationships. In these contexts, the presence of state authorities (or customary chiefs) in the room might affect what is being said and by whom. It is important to ensure safe spaces are created for debate and discussion. • Humanitarian and long-term needs: Communities in fragile and conflict-affected contexts often find themselves in dire material conditions. These pressing needs might strongly influence the content of a discussion about external assistance to these communities. • Psychological issues and trauma: Opening up conversations about sensitive topics might be very important for monitoring, but threatening to the psychological health of those consulted. Whenever possible, consideration of these issues should be central to the way NSM is used. • Language: The language used in the meeting will be a factor in who is able to take part. Similarly, using a nonlocal language might prompt participants to use “project speech,� reflecting the objectives of the intervener more than the realities of the communities. Step 3: Select the NSM methods and tools that address monitoring needs and fit the context The third step in designing NSM is to select the methods or tools. The choice of methods and tools has to be suitable for the project context. Some require more technical capacity than local actors may have. NSM can be done with tools such as in-depth key informant interviews and focus group discussions (FGDs), which are among the most frequently used tools. Most EISR+ efforts relied on qualitative rather than quantitative inputs in their assessments. Interviews and FGDs need to be carefully recorded, unless there are social-cultural and security constraints. The first main selection choice is between participatory, third party monitoring, or a combi- nation of both. Sometimes the NSM objective and focus area makes this choice clear. For Participatory and Third Party Monitoring in World Bank–Financed Projects  11 example, monitoring the targeting of local project benefits or the use of natural resources can- not be affordably done on a broad basis by an outside group. Textbook delivery, fishing rights, irrigation water allocation shares, and so on are best monitored by the user groups or com- munities that know the situation first hand. Therefore, participatory monitoring (or a combined method with strong emphasis on community feedback) may be the most feasible. On the other hand, if the primary purpose of NSM is to triangulate government data through rigorous verification, then third party monitoring becomes the clear choice. Some types of monitoring can become very technical—especially of budgets,7 expenditures, and procurement—and are usually performed by more specialized organizations. Table 2 describes the characteristics of common methods and tools for NSM that should be considered in selecting the tool. Table 2 Characteristics of Common Monitoring Tools used by Non-state Actors Government Citizen Technical Time Required Cost to Cooperation Participation Complexity and to Implement Implement the Monitoring Tool Required Required Skills Required the Tool Tool Focus group discussions Low High Low Low Low Citizen report card Low High High High High Citizen satisfaction survey Low High High High High Community scorecard Medium High Medium High High Social audit Medium High Medium High High Participatory physical audit Medium High Medium Low Low Procurement monitoring High Medium High Medium Medium Budget monitoring High Medium High Medium Medium Input tracking High Medium High High Medium Public expenditure tracking High Medium High High High Note: The classification of each NSM tool is subjectively based on typical applications and can vary depending on the specific application of each tool. There are six critical “C�s to consider in choosing NSM tools: context, cooperation, champi- ons, capacity, complexity, and cost. Each of these is useful in steering away from methods or tools that may be too difficult to implement given the circumstances. This will leave a set of feasible methods, but not point to a specific tool. The relative merits of NSM methods need to be evaluated on a case-by-case basis. The following characteristics, which expand on the six “C�s, are discussed below: • the overall legal, political, and social context for NSM • the extent an initiative depends on government cooperation • the need for champions and citizen participation • NSM capacity and experience among stakeholders • the complexity or difficulty of implementing the NSM activity • the time and cost of implementing a NSM tool 7. Not all budget monitoring efforts need to be technical. Basic aspects of a budget can be fairly easily under- stood and tracked by people/CSOs with minimal training and support. For example, a budget demystification exercise of local government budgets that was carried out in Bangladesh, in partnership with national media, was quite effective and found wide public resonance. 12  How-to Notes While there is no set order among the six criteria for selecting a monitoring tool, the choice is strongly affected by the context identified in Step 2, especially government attitudes towards NSM and social accountability more generally. In general, it is better to use simpler, less politi- cally contentious NSM tools in contexts that are not very conducive to NSM. For example, focus group discussions and interviews are common in all contexts and can provide a start. NSM methods might gain momentum through government cooperation and the support of champions, key individuals in government, or other organizations that can support NSM. The key is to select tools that do not exceed the extent of government cooperation required for them to work. For example, if NSM is trying to track budgets, inputs, or expenditures, that depends on government providing the necessary information. If government cooperation is reluctant, consider NSM methods that are less threatening, demanding, or challenging to government. For example, tools that provide insights into project implementation and service delivery such as community scorecards may work well. Consider working with local govern- ments and service providers that may be more open to NSM. If government cooperation is stronger, then more probing tools (that could reveal corruption) such as public expenditure tracking, procurement monitoring, and budget monitoring can be considered. Governments often challenge NSM based on its methodology. If that is a plausible risk, it may be necessary to complement perception-based feedback with other quantitative data, and to reach agree- ment at the outset on the kind of methodology to be used. Ultimately it is not just government tolerance for NSM, but willingness to act on the findings, that is critical. Some methods require much greater citizen participation to implement. Citizen participation might be much harder to mobilize in certain project contexts, due to illiteracy, political tensions, cultural (including gender-based) constraints, lack of knowledge, and lack of access to ICTs. Being aware of these factors would affect the choice of the tools. Participatory monitoring approaches are less feasible if citizen participation is constrained, but TPM can still work. Complexity refers to the level of technical expertise required to implement a NSM tool. The choice of tool can be further narrowed based on the capacity and experience required of NSM actors, both the monitoring organization and citizens. More complex tools may run the risk of marginalizing the poor and less educated. Most participatory monitoring uses simple methods for this reason. While some project areas might not have local organizations with the requisite skills to implement certain NSM tools, other organizations might be able to provide or teach the skills. More complex tools restrict the choice of implementing organization, such as independent budget analysis, procurement, public expenditure tracking, and procurement monitoring. This is particularly the case if the objective is less about community participation and ownership and more about independent data collection and verification. Another consideration is cost. NSM tools vary widely in the amount of time and resources required. Cost and time is affected by whether the tool is applied once, periodically, or continu- ously, and the number of locations. More complex tools generally require more financial and management resources. Citizen report cards, for example, rely on experienced organizations with specialized professional quantitative research skills. This takes significant time and money. In a sample of community scorecards, citizen report cards, and public expenditure tracking surveys, costs ranged from US$15,000 to $150,000 but can vary more widely. Most tools can be used for either participatory or third party monitoring, but some are more closely associated with one approach. Tools more often associated with participatory moni- toring include community scorecards, social audits, and participatory physical audits, among many others. Tools used mainly for third party monitoring include citizen report cards, procure- ment monitoring, input and expenditure tracking, and budget monitoring. They tend to require the higher capacity NSM actors typically associated with third party monitoring, particularly for data collection, analysis and reporting. Participatory and Third Party Monitoring in World Bank–Financed Projects  13 Some tools are more appropriate at a sector or national scale while others are more geared toward the local level. Community scorecards are more suitable for the local level where face- to-face and honest interaction is central to the tool. Citizen report cards or beneficiary sur- veys are better able to address larger populations through sampling methods and quantitative approaches. Some projects have used both or have created hybrid tools. There are no hard and fast rules on which tools to use. The choice depends on which tools are the most useful in addressing the specific information needs, as well as cost and time implications. It is beyond the scope of this note to provide many details about specific NSM tools. Annex 3 describes six common NSM tools, their key steps, main benefits and disadvantages, and implementation tips. There are other guidance notes on specific NSM tools such as com- munity scorecards and citizen report cards.8 There are many resources on NSM tools on the SDV Social Accountability database (FURL: “SAccdb�) and the Social Accountability E-Guide (FURL: “SAEGuide�). Role of ICT in Enhancing the Effectiveness of Monitoring Tools and Methods The use of information and communication technologies (ICTs) is revolutionizing non- state monitoring (NSM). ICTs can help make people more informed so they can provide more effective feedback and participate in citizen-based monitoring initiatives, increase inclusiveness and public outreach, and transform government-citizen relations. ICTs are useful in aggregat- ing individual voices and reducing the time and costs (including transportation costs) needed for participation in face-to-face meetings. Most NSM tools can use ICT for data collection and analysis, and translation of data into information. Therefore, ICT is not considered a separate tool, but a means for enhancing monitoring by NSAs. One ICT that is transforming NSM is mobile phones. They have been used to facilitate citizen feedback, submit grievances, and conduct data surveys. Smart phones, with their camera and GPS capabilities, have enabled NSM at the community level in a way that was not previously possible. One recent innovation is geo-tagging, which attaches location-specific information to websites, blog entries, photos, videos, SMS messages, and other media. Some projects have used mobile phones to take photos and send text messages about what is happening on the ground (see Box 10). Box 10 Using Mobile Phones in The objective of the West Africa Regional Fisheries Program is to increase sustainable in- Non-state Monitoring come through marine fisheries off the coast of West Africa. The project aims to reduce illegal fishing, put in place long-term policy reforms for the sustainable use of fisheries by the com- munities, and to give communities a stake in the management of and benefits from fisheries. As this is a high-rent and high-stakes sector in this region, monitoring of regulations (such as those to prevent illegal fishing) through the centralized government would not be feasible. The project strives to make communities responsible for designing their own strategies for long-term natural resource management and monitoring. Local committees report to the government and the national fishery association. A participatory monitoring system was piloted in Liberia to track illegal fishing. The communities were trained to track a few indicators. They also used mobile phone photos and text messages to document the presence of illegal fishing boats off their coasts. Ten communities participated and the information was centrally uploaded to a website. GPS coordinates were attached to illegal boats, and mapped. (continued) 8. Two examples are “Rapid Feedback: The Role of Community Scorecards in Improving Service Delivery� World Bank (2011b), and “Citizen Report Cards: Monitoring Citizen Perspectives to Improve Service Delivery� World Bank (2012). 14  How-to Notes Box 10 (continued) The World Bank contracted International Relief and Development (IRD), a U.S. NGO, to provide third party monitoring of the Afghanistan Reconstruction Trust Fund (ARTF). Given the security challenges and widespread corruption in Afghanistan, the Bank wanted a monitoring mecha- nism that would reliably report on the construction of village-level infrastructure in areas that were difficult to visit. IRD field engineers are outfitted with smart phones with data collection applications (apps) developed by IRD for each type of ARTF project, such as buildings, roads, bridges, and canals. The individual project survey apps also include social and environmen- tal questions. The apps also allow the field engineer to record observations, interviews, and information about the project, which can later be translated as necessary. At the site, the field engineer uses the smart phone and application to collect required data as well as take geo- referenced photos. When finished, the data is transferred via a cellular data network or the In- ternet to an online data processing system; there it is checked for accuracy and completeness before uploaded into the ARTF database. The online project catalogue contains supporting documents such as inspection reports, drawings, photos, and so on. This provides a compre- hensive database of project implementation and performance. The Tanzania Secondary Education Development Program aims to improve the quality and relevance of education in secondary schools through government capitation grants to schools. To monitor the government delivery of capitation grants, a coalition of local NGOs use mobile phones to collect data from headmasters. An implementing NGO, Twaweza, called 50 randomly selected headmasters in 14 regions to verify receipt of the capitation grant. The NGO also used mobile phones to monitor road blocks and bribes that truckers faced. It asked 25 truck driver volunteers to record where they were stopped by which authority, for what reason, for how long and whether a fine or a bribe was paid and the amount of a fine/bribe. A toll-free number was established by the implementing NGO to allow drivers to call or “beep� the NGO and for the NGO researchers to ask questions and collect data. Mobile phone surveys have collected data and information on a wide variety of issues including health, education, nutri- tion, water, security, travel times, prices, electricity, and governance. The choice of ICT tools and approach usually is determined by following factors: • access to technology (for example, according to gender, income, education, and age) • location and distribution of monitoring sites (including number, distance, security, and access) • cost and time required to implement the ICT • purpose and complexity of NSM activity (for example, complexity and length of the questionnaire) • monitoring organizations’ and citizens’ interest and capacity to use ICT Step 4: Decide on implementation details for NSM (methods, actors, budgets, and so forth) The fourth step is to decide on the practical implementation details such as who, how, the scale, budgeting, and so on. This is the heart of designing NSM. Key questions include the fol- lowing: Who will collect the data and from whom? Will it be done jointly with government? How will it be reported? Who will be using it? How and when will it be used? Stakeholder analysis and other research might be necessary to answer these questions. This note provides only a set of issues to consider and some practical advice. There is no fixed set of steps to follow, but most NSM has to address the issues discussed in this section. Based on experience, one key principle is simplicity (see Box 11). Gathering data and information. NSM is more than a tool. What matters is how it is used. The primary issue is collecting the kind of data that can be processed, organized, and inter- preted as meaningful information. Questions need to be phrased in a way that people can understand and provide reliable responses. Key questions to be considered before developing questionnaires and collecting data include: Participatory and Third Party Monitoring in World Bank–Financed Projects  15 Box 11 Tips for Designing Non-state In general, it is better to err on the side of keeping NSM simple rather than too complicated. Monitoring: Keep it Simple There is often a tradeoff between the quality and sophistication of monitoring versus capacity building and ownership of the process by involved stakeholders. Too often the Bank creates NSM designs that look good on paper but prove very difficult to implement. Therefore, the fol- lowing tips are recommended: 1. Assess existing capacity before beginning the NSM design so it matches existing capacity. 2. Keep monitoring designs flexible and simple in projects with limited civil society capacity. 3. Do not let the design be too Bank-driven, too complicated, or too superficial in understand- ing the local context. 4. Allow more time for implementation than estimated. 5. Use a simple structure of what to report on and who to report to, with very clear roles. 6. Aim for some early successes to keep NSM actors committed. • What kind of information is required? • What kinds of methods will be used to obtain it? • How will the indicators be determined and by whom? • What are the key indicators that NSM should focus on? Information could be about services communities are supposed to receive from the project, positive and negative impacts of the project, and ways of improving project implementation and its impacts. The desired information goes far beyond indicators, which have a limited abil- ity to capture what matters to many stakeholder groups and are often inadequate at explaining causes. Therefore, the focus should be more on the needed information than the indicators. One method is to ask project stakeholders to list their most pressing concerns as a means to identify relevant indicators and issues. This also helps develop a shared understanding of indicators among various actors. Reporting. An important planning step is deciding on how to report the information collected through NSM. There are numerous cases where the data collected has been conveyed by the monitoring actor in long descriptive reports, lacking key conclusions and analysis. One way to minimize this risk is to agree on the reporting formats as part of detailed terms of reference9 and provide clear guidance and reporting templates to the monitoring actor. Scale and representativeness. Another choice is how broadly monitoring will be applied. NSM can be at the national, regional, district, local, or community level. It can be piloted in a few areas or used project-wide. Resource limitations usually require a trade-off between broad sampling and intensive NSM in a few locations. Whatever choice is made, the data needs to be sufficiently representative to derive reliable conclusions about project performance and needed actions. Timing. NSM is preferably an element of the project from its design, so that it becomes a part of the project’s DNA and is properly resourced. It is possible to integrate it later on (good opportunities are during midterm reviews or restructuring), but experience has shown that a later start might leave less room for maneuvering. NSM can also take place toward the end of the project for assessing beneficiary satisfaction and as part of project evaluation. Another tim- ing issue is the frequency of NSM: Will it be a one-time exercise, periodic, or ongoing through- out the project? Timing and frequency issues depend on the purpose of NSM, the methods used, and the available resources. 9. For reference, please see sample Terms of References in the Social Accountability database, under the “Third Party Monitoring� tab (FURL: “SAccdb�). 16  How-to Notes Selecting the monitoring organization. This is clearly one of the most important decisions. The capacity and skills of the monitoring organization must be sufficient to implement the NSM. Participatory monitoring requires the capacity to mobilize NSM participants. Capacity requirements will narrow down the list of potential organizations. It is highly desirable that the organization not only has experience on the issue or methodology, but also an in-depth coun- try- or region-specific knowledge. This is many times overlooked in NSM of Bank projects. The NSM organization should have a reputation for integrity, competence, and credibility so that the findings will be accepted. Good reputation also plays a role when this organization is “paid for� by the project. The organization’s relationship with the government, project, benefi- ciaries, and communities has to be good enough that it is respected, trusted, and accepted. However, for third party monitoring, the relationship should not include full dependence on the government or project that might jeopardize the third party’s ability or incentive to provide an independent perspective. In all cases, the Bank teams need to watch out for the risk of co- optation (see Box 12). Box 12 Selecting Partners for Burkina Faso has a relatively active civil society but it remains poorly coordinated and highly Third Party Monitoring in politicized. There are a handful of national research and advocacy-based think tanks that have substantial financial support from donors on issues regarding electoral processes, corruption, Burkina Faso and human rights. Despite their prominence, these entities have limited influence and con- tact with people outside urban centers and the capital. Rural producers, women’s, and youth groups tend to lack access to information, financing, and power. Access to financing is a major constraint that leads to competition for financing, lack of specialization, intense rivalries, and diminished effectiveness of grassroots organizations and increased susceptibility to corruption. In this challenging environment, two different arrangements were tested. In the agriculture and transport projects, a for-profit firm was selected on the basis of its capacity to coordinate and conduct qualitative assessments of this kind. In the case of the water sector, an umbrella na- tional CSO with an existing involvement in sectoral dialogue was selected. Capacity building. Often, capacity building is needed for one or more NSM actors. For example, CSO capacity is often concentrated in the capital city. CSOs in the project area may need training. Determine which actors need capacity building and in which areas. This could begin with identification of core information needs. Capacity building could require training on the World Bank’s rules and regulations (especially on reporting, procurement, financial man- agement, and payment rules), formulating relevant and feasible indicators, benchmarking, data collection and analysis, or how to use findings for reflection and to identify corrective actions. Budgeting. The budget for NSM itself may be self-evident, but there are several associated costs. Bank staff time for providing support for NSM tends to be underestimated, especially in low-capacity and fragile environments. This is in addition to the budget required for training and capacity building. Quite often, NSM requires facilitation, feedback strategy, and tracking the performance of the monitoring organization. The costs of scaling up and institutionalization, if that is planned, need to be factored in. Funding sources. Project budgets are generally the largest source of funds for monitoring. Using project funds also has the advantage of embedding NSM in a project component. However, government may not be keen on using project funds for non-state actors they may be seen as watchdogs over the government, or too burdensome, or unnecessary. If project funds are insufficient, trust funds can help fill the gap. Some trust funds that have been used are the Governance Partnership Facility (GPF), Japanese Social Development Fund (JSDF), State and Peace Building Fund (SPF), and Institutional Development Fund (IDF). Strategic partnerships have become increasingly important in addressing potential conflicts of inter- est and investing in longer-term monitoring capacity of non-state actors. The Bank’s Latin American and Caribbean Region signed the first Strategic Partnership on Transparency, Social Accountability, and Governance with USAID to enhance CSO capacity to monitor projects and programs. In Africa, similar partnerships have been agreed with the U.K.’s Department for Participatory and Third Party Monitoring in World Bank–Financed Projects  17 International Development (DfID), the Swiss Agency for Development and Cooperation (SDC), and others. The new Global Partnership on Social Accountability (GPSA) is expected to fund third party monitoring among other social accountability activities. If trust funds are used, gen- erally there is more flexibility and less pressure to obtain buy-in from ministries or project man- agement units. However, NSM might not be as well integrated into the project, or the findings might not be utilized. Funding and independence. One more funding issue is the independence of NSM and conflict of interest (see Box 13). If the project is paying for the NSM, it can make the monitoring organization beholden to it and NSM is not truly independent. One can increase independence by using nonproject funding. Funds for NSM are often difficult to obtain; thus, using an indepen- dent funding source may not always be a viable option. Most monitoring by non-state actors in Bank projects is paid for by the project or the government. However, it is more important to select the right organization than to be too “purist� about independent funding. The reputation, credibility, and track record of the monitoring organization is often the most important measure of its independence. Box 13 Funding Third Party When Bank-financed projects are monitored by third parties, the monitors are often contracted Monitoring through an and paid for by the same government agencies they monitor. This begs the question: can the monitors be truly independent? A pilot program in the South Asia Region aims to mitigate this Independent Entity: conflict of interest and increase citizen engagement. It channels trust funds for project monitor- Bangladesh and Nepal ing not through the implementing agencies but through an international NGO, the Partnership for Transparency Fund (PTF). PTF is a leading NGO supporting CSOs and citizens in fighting corruption through monitoring and mobilizing citizens to assert their rights. National CSOs in Bangladesh and Nepal help the Bank, national governments, and PTF identify local CSOs to re- ceive grants for monitoring activities through a competitive process. PTF is using a US$1.9 million grant from the Japanese Social Development Fund to select, train, and fund CSOs to monitor 25 Bank projects. Through its network of local partners, PTF advises the CSOs as they monitor the projects. The programs will last three years as the grants ranging from US$25,000 for a local pilot to US$150,000 for a national program are awarded in tranches. Integrating NSM into the project. NSM is not likely to be very useful if it is not well integrated into the project. The findings may not be used and it will be a wasted effort, even if NSM is done well. The key is to make NSM integral to the project and its success. One way is to make NSM findings relevant to project decision making and performance. Link NSM to regular internal monitoring in the project and the project results framework (see Box 14). If NSM was not part of the original project design, making this link can take time. If NSM is focused on Box 14 Integrating Feedback into The objective of the Bangladesh Local Government Support Project (LGSP) is strengthening of Government Systems: local government through systemic improvements to governance, public administration, and service delivery. Feedback is collected at multiple levels: neighborhood, ward, local govern- Bangladesh Local ment and higher. Beneficiary feedback collection is mandatory. Feedback is evaluated and Government Support used for calculating and distributing performance-based intergovernmental fiscal transfers in Project the form of block grants. Other measures complement beneficiary feedback. Scheme Supervi- sion Committees and Scheme Implementation Committees enhance people’s participation in priority setting and monitoring. Social accountability tools are used to help demystify information and channel feedback from citizens to public officials. Social audits and community scorecards were piloted in 50 Union Parishads (UPs, local government councils) in Dhaka Division during 2010–11. A local journalist network has been supported to collect beneficiary feedback through different media and report it: 63 journalists and 64 community leaders were trained in social audit methods and facilitated pilot social audits in the UPs. A toll-free number has been estab- lished to receive feedback nationwide 24/7 via phone calls and text messages. A decentralized national monitoring system has been established by the local government ministry for the first time in the country to collect record and respond to beneficiary and other forms of feedback. 18  How-to Notes government performance, it needs to be integrated into government monitoring systems or it will likely be ignored. Ongoing dialogue with and active participation by government officials in NSM activities improves the likelihood that project implementers and decision makers will value and utilize the findings from NSM. Implementation This section elaborates a four-step process to help project teams think through NSM imple- of Non-state mentation in their projects. Figure 3 portrays the four main steps with the main actions listed below each step. Monitoring Figure 3 Steps in Implementing 1. Pilot Methods 2. Analyze Data 4. Sustain (and 3. Track, Refine, and Monitoring by and Test the and Validate Findings Document NSM Scale Up) the NSM Process Initial Design with Stakeholders Non-state Actors • Conduct first round of • Analyze data • Monitor, refine, • Institutionalize NSM • Share and discuss and document state-citizen relationships • Test assumptions about findings with implementation • Promote close links and the NSM design stakeholders • Translate NSM communication flows • Obtain feedback from • Validate findings and instruments, manuals, among NSM actors NSM users and reflect on implications of and other materials into implementers the findings appropriate languages • Use feedback to revise • Develop action plans • Ensure that NSM findings NSM design as needed based on findings are used to adjust the project/program Step 1: Pilot field methods and test the initial design Introducing NSM requires a period of adjustment to work out the details. Usually it is a new experience and people are not sure what to expect. It is best not to rush the process. Everyone needs to understand their roles. The first round of NSM is a testing period to see how it works and if it is ready for full-scale use. This is when a lot of the assumptions during NSM design will be put to the test. Even at this early stage it is vital to obtain feedback from the main users and implementers of the NSM initiative before full rollout. Pilot testing can identify any additional needed capacity building. This is the time to aim for some early successes to strengthen commitment to the process. NSM should not be seen as a one-off event, but as an evolving process. Step 2: Analyze data and validate findings with stakeholders Once the NSM is under full implementation, the data needs periodic interpretation and analy- sis. The monitoring organization can do the first cut of data analysis, but then it helps to bring other stakeholders into the analysis. Which stakeholders are involved depends on the project, but they include the government, beneficiaries, communities, project-affected people, and others with a stake in the findings. When possible, Bank staff should also participate, including financial management, safeguards, and relevant sector specialists. Stakeholder participation is needed in interpretation, processing, and analysis of the data, especially for participatory monitoring. Stakeholders engage in critical reflection on project performance, problems and constraints. The audience determines how the data should be presented in a format they can easily understand. Documentation should be simple, clear, brief, timely, and accessible. Then the stakeholders need to discuss how the findings will be used. Sometimes this includes an action plan to address any issues that NSM revealed. Data analysis and action planning should not be seen as threatening, antagonistic, or fault-finding, but rather as constructive engage- ment to find solutions. Step 3: Track, refine methodology, document, and utilize the findings of NSM NSM itself needs to be monitored to identify implementation problems. Supervising NSM often takes more time than supervising regular monitoring because the monitoring is not done by the project team or government agency. The extra time required needs to be planned for. The Participatory and Third Party Monitoring in World Bank–Financed Projects  19 Box 15 What Happens After Monitoring data is collected by field engineers using smart phones and data collection appli- Collecting the Data? cations developed by International Relief and Development (IRD). The data is used to produce monthly inspection reports and quarterly analytical reports identifying issues that need address- Afghanistan Reconstruction ing. The data and findings are discussed in monthly meetings between IRD, the Bank, and gov- Trust Fund (ARTF) ernment ministries. Quarterly meetings with ARTF management examine cross-cutting issues. Joint missions go to the field and a supervisory agent participates in Bank supervision missions. These reports, meetings, and joint activities facilitate data analysis, validation, and utilization. project implementation unit and Bank team need to actively guide the monitoring organization. This includes refining the methodology (and advising on what information would be useful), methods of analysis, and reporting formats. Although NSM has been used to monitor projects that have numerous, widely dispersed, or conflict-affected sites, implementation can reveal limitations that require adjustments. For example, in the ARTF, the number of proposed sites for monitoring was overly ambitious and had to be scaled back. An important part of the process is ensuring communication flows among stakeholders so that the findings are used. Monitoring by itself does not improve project performance unless there is clarity and continuity in the monitoring, and in the working relationship between the moni- toring agency, the government, and project staff. Therefore, it is critical to develop feedback channels from government to communities, project beneficiaries, and others, and to ensure that feedback is used to change project processes. The Bank can serve a bridging function and promote NSM and using the findings. Some ways to help ensure that NSM findings are utilized include: • Regular meetings between the government and the non-state actor to discuss monitoring findings • Discussing necessary actions and possible remedies to address the problems across the project • Joint missions so everyone learns from each other on what to look for during site visits • Training on project monitoring, report writing, and how information from project monitoring should feed into decision making and inform possible changes to project design NSM also needs to document information about the NSM process itself and to assess its impacts to determine the value added of NSM. One needs to be realistic about how much NSM can achieve in the short run, especially in terms of fostering accountability. This requires a long-term agenda, since it involves measuring changes in attitudes, values, behaviors, and incentives. Step 4: Sustain (and scale up) the NSM initiative Sustaining NSM is a big challenge. The activities covered under non-state monitoring should be designed to institutionalize relationships between the government and citizens as much as possible over the long term, rather than “projectize� relationships. Some ways this can be done include: • Promote close links between government officials, media, CSOs, and communities • Support ways of increasing the comfort level and recognition of monitoring done by non- state actors in ministries and the project management units • Support networking, experience-sharing, and peer support among non-state actors • Tailor manuals and other materials to leave behind beyond the pilot monitoring initiative • Share results with the public in a way that is meaningful to their everyday realities and that engages them in their own spaces 20  How-to Notes Task managers are encouraged to attach NSM reports to supervision mission aide memoires and implementation supervision reports (ISRs). Sharing findings with a broader audience— for example, through a dissemination workshop and a website to make reports accessible online—can increase NSM impact and build support for sustaining it. The pilot NSM can be passed on to high-capacity CSOs. Finally, many NSM initiatives are not intended to be scaled up, but if the pilot is successful enough and there are sufficient reasons for expanding it, then the team may consider several factors. Is the existing initiative appropriate for scaling up? Are any changes necessary for scaling up? That is why documentation of NSM is necessary, to understand what is working and why, so any shortcomings can be corrected before scaling up. What works in one loca- tion might not work in other locations with other stakeholders. The original initiative may need adaptation and other adjustments. What additional burdens will scaling up place on NSM actors? They may need capacity building. New actors may also be needed. Tips for Working with Beyond the steps outlined above, this note offers the following tips for implementing NSM. Different Actors Besides lack of capacity for NSM, some of the most common challenges in implementing NSM are government or project management unit resistance or indifference to NSM, lack of Involved in NSM feedback to monitoring outputs, lack of clearly defined roles and responsibilities, and a ten- dency to be superficial or “tokenistic� rather than fully committed to NSM. These tips are mainly about the politics, incentives, and institutional relationships that underpin successful NSM. In the long run, these are more critical to success than getting the techniques or tools right. Tips for working with non-state actors. Be realistic about what CSOs can achieve. Often they lack capacity, but even when they have sufficient capacity, introducing NSM in apprehen- sive or indifferent environments requires changing the institutional culture of government or other stakeholders. This is beyond the capacity of non-state actors or any one NSM initiative. But combined with government internal reform and other efforts, NSM can contribute to a more conducive environment for NSM, mainly through successful pilots that demonstrate its utility. Therefore, allow sufficient time for consultation and training, not only on technical moni- toring methods, but Bank procedures and processes, procurement, and reporting formats and requirements. Be prepared to provide implementation support along the way. Opting for a gradual rather than a “big bang� approach can pay off over the long run. To be effective pro- moters of good governance and accountability, CSOs often need to rethink and, sometimes, reform their own internal governance and accountability. Tips for working with government. Governments may see NSM as an extra level of moni- toring that creates more work, may not be worth the effort, and might not be in their interest. A proactive approach is often needed to build interest and commitment for NSM. Therefore, NSM should be presented as identifying and managing project risks to improve government performance rather than as a watchdog. It helps to demonstrate the value of NSM with practi- cal examples. Identifying and working with champions can play an important role. One practical measure to improve government buy-in is to highlight the positives as much as the negatives. Governments appreciate positive feedback. Ultimately, if governments do not have the willing- ness and ability to respond to NSM findings, NSM cannot accomplish much. Therefore, NSM often needs to be complemented by measures to increase government capacity, awareness, understanding, and appreciation for NSM to enhance project effectiveness. Getting government and non-state actors to work together. The World Bank has a com- parative advantage in convening different stakeholders and should perform this role, but it needs to tread lightly. Building the relationship between government and non-state actors is crucial for success. For example, the Bank can support formal cooperation mechanisms such as memorandums of understanding (MOUs) between government and CSOs that outline Participatory and Third Party Monitoring in World Bank–Financed Projects  21 the terms of engagement and expectations for government response and feedback to CSO monitoring. It is best if project teams can share findings with the beneficiaries as part of closing the accountability loop. Capacity building on NSM should involve government officials that are working with the non-state actors. Greater presence of Bank staff in the field facilitates ongoing dialogue between government and non-state stakeholders, and promotes better implementa- tion. Different stakeholders may have very different objectives, assumptions, experiences, and attitudes about NSM, so communication is necessary to understand and bridge these differ- ences. Sometimes non-state actors are hired that do not adequately understand the project and this further exacerbates tensions between them, project management units, and Bank staff. They often fail to see implementation issues from the beneficiary, Bank, and government perspective. Understanding the organizational culture of partner organizations and their experi- ence with NSM is the key (see Box 16). Box 16 Developing Government- In the Bangladesh Social Investment Program Project (SIPP), the biggest challenge was tension Civil Society Relationships between project staff and the monitoring agency, as the staff felt threatened by the agency reports. This was true of both Bank staff and the project management units (PMUs). It took about through Monitoring: The two years to build relationships and change the culture of project staff so that monitoring Case of Bangladesh agency and project staff could work together. Regular monthly meetings, where everyone sat together and discussed issues, were the key to achieving this. This relationship made the PMUs more responsive and less defensive. The project staff then perceived the monitoring agency as not policing, but as helping troubleshoot. Initial training and technical assistance proved criti- cal. In this type of process monitoring, it is important that the monitoring focuses on process as well as results. Monitoring by itself does not improve project quality unless there is continuity in the monitoring and working relationships between the agency and project staff. Tips for working with communities. It is important to foster relationships between outside stakeholders and the community that is either involved in the NSM or is a primary subject of monitoring. Participatory monitoring needs to build local commitment to succeed. An entire community cannot be involved because that would be too labor- and time-intensive. Usually a committee represents them. These community committees should include men, women, youth, marginalized, and vulnerable groups (sometimes in separate meetings). Elite capture or undue influence by traditional authorities is often a problem, and thus there are increasing efforts by Bank teams to engage a larger number of beneficiaries directly through face-to-face dialogue or ICTs. Local institutions and NGOs can be used to support monitoring committees. However, these NGOs need to be carefully screened and selected on the basis of experi- ence, skills, local presence, and legitimacy in the affected area. Train community members in project monitoring, reporting, and communication skills. It is better to start slowly so people can become comfortable in articulating their views. Remember to keep their perspective and preferences in mind. For example, adjust to their schedules and arrange meetings in venues that are suitable to them. High-quality community facilitators are vital. Communication is critical. People need to be informed about the project in order to under- stand what is expected. Periodic information sessions and public disclosure of project infor- mation helps raise awareness. Keeping communities informed about findings and how their feedback is being used will help maintain their interest and involvement. Working with the media and other communication experts increases the impact of monitoring through public aware- ness. Making findings public creates the most leverage. However, it is important to consider how much information should be shared and with whom. While there will inevitably be a lot of pres- sure to share all information with all stakeholders, some of the information might be quite sensi- tive and should be handled with care. Not all findings should be shared with all stakeholders. 22  How-to Notes Key Takeaway In summing up, this note shows how non-state monitoring can be a useful addition to internal Messages project monitoring for a wide variety of reasons. It provides a more comprehensive picture by recording information and perspectives that standard monitoring may fail to capture. The monitoring approach depends largely on the information needs, objectives, and the project context. The presence or lack of technical capacity shapes the feasible approach and choice of NSM actors and methods. NSM is much more political than technical. It is critical to have a firm understanding of the poli- tics and institutions in the project context to guide NSM design and implementation. As a long- term process, NSM needs to be implemented with patience, commitment, and resources. Designing effective communication flows and processes increases the chances for success. References Darendeliler, Zeynep. 2012. “Monitoring through Beneficiary Feedback: A Stocktaking of Monitoring by Non-State Actors in World Bank–funded Projects.� Social Development Department. World Bank, Washington, DC. World Bank. 2011a. “How, When and Why to Use Demand Side Governance Approaches in Projects.� Social Development-GAC in Projects Guidance Note. World Bank, Washington, DC. ———. 2011b. “Rapid Feedback: The Role of Community Scorecards in Improving Service Delivery.� How-To Note, Social Development Department. World Bank, Washington, DC. ———. 2012. “Citizen Report Cards: Monitoring Citizen Perspectives to Improve Service Delivery.� How-To Note, Social Development Department. World Bank, Washington, DC. ———. 2013. “Enhancing Transparency Accountability through Citizen Feedback: Lessons from the E-ISR+ Pilot. Africa Operations Services Series, The Africa Region. World Bank, Washington, DC. http:// documents.worldbank.org/curated/en/2013/03/17429966/enhancing-transparency-accountability- through-citizen-feedback-lessons-external-implementation-status-results-plus-pilot. Participatory and Third Party Monitoring in World Bank–Financed Projects  23 Annex 1 Choices in Non-state Monitoring (NSM): Approaches and Tools Choices in NSM Design NSM Objectives NSM Priority Generally Preferred Type of Monitoring Purpose of NSM Independent perspective on project performance Third party monitoring Beneficiary empowerment Participatory monitoring Focus of NSM Project progress, outputs, outcomes, etc. Third party monitoring Stakeholder perceptions, etc. Participatory monitoring NSM Context Type of Context for NSM Potential NSM Tools Citizen and civil society interest High level of interest in NSM Community scorecard in NSM Social audit Participatory physical audit Medium level of interest in NSM Expenditure or input tracking Budget or procurement monitoring Low level of interest in NSM Citizen report card Citizen satisfaction survey Government attitudes towards Government is more receptive to NSM Expenditure or input tracking NSM Budget or procurement monitoring Community scorecard Government is less receptive to NSM Citizen report card Citizen satisfaction survey Participatory physical audit Capacity of NSM actors High level of technical capacity for NSM Expenditure or input tracking Citizen report card Citizen satisfaction survey Medium level of technical capacity for NSM Community scorecard Social audit Participatory physical audit Low level of technical capacity for NSM Interviews Focus group discussions Resources and time available High level of time and resources for NSM Expenditure or input tracking for NSM Citizen report card Citizen satisfaction survey Community scorecard Medium level of time and resources for NSM Social audit Budget or procurement monitoring Low level of time and resources for NSM Participatory physical audit Interviews Focus group discussions What is Being Monitored Monitoring Approach Potential NSM Tools Construction, infrastructure Third party monitoring Project quality monitors Participatory monitoring Participatory physical audit Beneficiary satisfaction with Third party monitoring Citizen report card project, service delivery, etc. Participatory monitoring Citizen satisfaction survey Community scorecard Budgets, procurement, Third party monitoring Budget or procurement monitoring expenditures, etc. Participatory monitoring Expenditure or input tracking Social audit 24  How-to Notes Annex 2 Non-state Monitoring (NSM): Steps in Design and Implementation Designing Non-state Monitoring 1. Identify the objectives, focus, and target audience of NSM: • Identify the broad objective (improve project effectiveness, obtain beneficiary feedback, etc.). • Assess existing monitoring and accountability chains for gaps that NSM could fill. • Identify the information needed to meet the monitoring objective. • Identify the primary focus (project inputs, processes, outputs, impacts). Identify the pri- mary target audience (project implementation unit, government, Bank, others). 2. Assess the context for NSM: • Review previous experience with NSM in the country and project area. • Assess political factors: government attitudes about and capacity for NSM, timing. • Assess social factors: citizen, CSO, and other non-state actor capacity for NSM, cultural norms about participation, accountability, and transparency. • Assess the implications of the location of the project: which non-state actors have suf- ficient access to the area, security of the project area, location of project sites, seasonal consideration for travel. • Assess capacity and security constraints, and beneficiary sensitivities in FCS areas. • Identify existing or conduct new analysis on the context. 3. Select NSM methods and tools that address monitoring needs and fit the context based on: • Third party monitoring, participatory monitoring, or both approaches are used • The overall legal, political, and social context for NSM • The extent an initiative depends on government cooperation, champions, and citizen participation • NSM capacity and experience among stakeholders • The complexity or difficulty of implementing the NSM activity • The time and cost of implementing a NSM tool 4. Decide on implementation details for NSM: • Decide what information is needed and what kinds of methods will be used to obtain it. • Decide on the key indicators, how the indicators be determined, and by whom. • Decide on the scale (how many locations) and frequency of NSM. • Select the organization that will conduct the NSM (capacity, reputation, relationships, etc.). • Determine capacity-building needs, how they will be provided, etc. • Determine the budget for NSM (including facilitation, training, supervising NSM). • Assess any conflict of interest issues between NSM implementers and NSM funding sources. • Integrate NSM into the project through feedback loops, decision-making processes, management information systems (MIS), etc. Implementing Non-state Monitoring 1. Pilot test and test the initial NSM design: • Conduct first round of NSM. • Test assumptions about the NSM design. • Obtain feedback from main users and implementers. • Use feedback to revise NSM methodology as needed. • Identify any additional needed capacity building. • Aim for some early successes to strengthen commitment to the process. 2. Conduct analysis and validate findings with other stakeholders: • Document the findings. Be simple, clear, brief, timely, and accessible. • Present the data in a format that suits the audience and purpose of the data. • Share and discuss findings with stakeholders, their implications, and how findings will be used. Participatory and Third Party Monitoring in World Bank–Financed Projects  25 • Develop action plans based on findings. 3. Track, refine and document NSM: • Monitor and supervise NSM implementation. Allow extra time for NSM monitoring. • Refine the NSM methodology, information needed, methods of analysis, and reporting formats. • Translate NSM instruments, manuals, toolkits, and other materials into local languages. • Ensure that NSM findings are used to take necessary actions in the project. • Document the NSM process and assess the value added of NSM. 4. Sustain the NSM process: • Promote communication flows between government officials, media, CSOs, and communities. • Support ways of increasing the comfort level and recognition of government NSM supporters. • Support networking, experience-sharing, and peer support among NSM practitioners. 5. Scale up the NSM initiative: • Decide if the existing initiative is appropriate for scaling up. • Decide if any changes are necessary for scaling up. • Assess what additional burdens scaling up will place on involved actors. • Decide if they need capacity building. Improving Relationships among NSM Actors 1. Working with non-state actors: • Be realistic about what non-state actors can achieve. • Allow sufficient time for consultation and training. • Be prepared for capacity building and support. 2. Working with government: • Present NSM as troubleshooting project risks and problems rather than as a watchdog. • Use NSM to improve government performance rather than criticize it. • Demonstrate the value of NSM with practical examples. • Identify and work with champions. • Align incentives to convince service providers about the benefits of NSM. • Consider working with local governments that may be more open to NSM. 3. Getting government and non-state actors to work together: • Focus on building the relationship between government and non-state actors. • Capacity building on NSM should include government officials and non-state actors. • Understand the organizational culture of partner organizations and their experience with NSM. 4. Working with communities: • Community committees should include men, women, youth, and marginalized and vul- nerable groups. • Use local institutions and NGOs to support monitoring committees. • Train community members in project monitoring, reporting, and communication skills. • Start slowly so people can become comfortable in articulating their views. • Adjust to community members’ schedules, for example by conducting meetings on weekends or in the evening. • Hire high-quality community facilitators. 5. Improving communication: • Build public understanding for NSM. • Conduct information sessions and disclose project information to raise awareness. • Keep the communities informed about findings and on how their feedback is being used. • Work with the media and other communication channels to increase the outreach of NSM. • Consider what information should be shared with which stakeholders. Annex 3 Six Common Monitoring Tools Used by Non-state Actors Objectives/Characteristics Steps Benefits/Uses Limits/Challenges Implementation Tips 1. Focus groups discussions are usually organized with specific goals, structures, time frames, and procedures. Focus groups are composed of a small number of stakeholders to discuss project impacts and concerns and consult in an informal setting. They are designed to gauge the response to the project’s proposed actions and 26  How-to Notes to gain a detailed understanding of stakeholders’ perspectives, values, and concerns. • To provide a smaller and more 1. Determine whom to invite • For sensitive issues and • More time consuming • Use a professional focus group intimate setting enabling and how. topics than larger public facilitator who is skilled, neutral, and stakeholders to express their 2. Identify open-ended • Allows different meetings knowledgeable about the topic. views more freely questions to ask. categories of • Some members of the • Manage expectations of stakeholders • To brainstorm or test possible 3. Explain the objectives stakeholders to voice group may dominate and clearly explain what the focus objectives and scenarios of consultations, rules of their views the issues brought up group discussion can and cannot do. among a cross-section of participation, and proposed • Facilitates exploration of • Not statistically • If stakeholders are illiterate, spend time stakeholder interest groups follow-up at the beginning of issues or concerns that representative to explain the project and its relevant the discussion. are not widely known or • Cannot generalize aspects in a language they can understood views and statements understand. 4. Agree on the agenda and invite participants to • To contextualize results of to represent wider comment on it. surveys and quantitative population data 2. A community scorecard is a community-based monitoring tool that assesses services, projects, and government performance by analyzing qualitative data obtained through focus group discussions with the community. It usually includes interface meetings between service providers and users to formulate an action plan to address any identified problems and shortcomings. • Uses the community as the 1. Identify the scope of the • Encourages local • Interface meetings • Find a champion to support and sustain unit of analysis scorecard. problem-solving through need to be well community monitoring teams. • Conducted at the local facility 2. Train resource persons. development of joint managed to avoid • Select skilled facilitators and give them level action plans increasing frustration a small stipend for their work. 3. Initiate an awareness and conflict • Used more in rural settings campaign. • Empowers service users • Advertise well in advance of the to express their needs • Better for rural settings intervention (through radio, field visits, • Generates information through 4. Develop an input tracking and opinions regarding (greater sense of posters and flyers) to ensure community focus groups matrix. the access and quality community) awareness. • Improves communication 5. Perform community scoring of services • Hard to sustain • Publicize the monitoring results between communities and of performance. • Encourages • Small-scale widely using various media to service providers 6. Conduct self-evaluation by accountability of service engagements with a sustain community interest and raise • Provides immediate feedback service provider staff. providers wide gap with national awareness on value of the process. to service providers 7. Hold interface meeting • Generates benchmark practice and policy • Repeat the community scorecard • Encourages local problem between community and performance criteria that processes exercise to monitor progress over time. solving facility staff. can be used in resource • Less emphasis on • Reforms are decided through 8. Conduct follow-up activities allocation and budget rigorous quantitative mutual dialogue such as analysis, advocacy, decisions data about users’ and dissemination of policy satisfaction rates changes. (continued) Objectives/Characteristics Steps Benefits/Uses Limits/Challenges Implementation Tips 3. A social audit is a monitoring process through which project information is collected, analyzed, and shared publicly in a participatory fashion. Social audits may go beyond the oversight of project finances and procurements to examine all aspects of the project including level of access to information, accountability, public involvement, and project outputs and outcomes. Social audits are typically carried out by community volunteers (social audit teams or committees) and findings are presented at a public forum or hearing. • Monitors the effects of, and 1. Define the scope of the • Enhances participation • Requires substantial • Social audits often benefit from the informs policy makers about, audit. of stakeholders, technical support, involvement of NGOs for training, public service delivery and 2. Identify stakeholders. especially of the poor especially in obtaining information, documenting findings, and local governance and marginalized and analyzing data follow-up with officials. 3. Initiate an awareness • Assesses the views of citizens campaign. • Trains community on • Crucial public access • Be clear that the public hearing is not a about public services, participatory monitoring to records often finger-pointing exercise, but rather that measures citizens’ knowledge 4. Choose indicators and depends on political public discussions are an opportunity • Promotes accountability about local governance consultation process. willingness for the local community to provide and transparency • Increases informed interaction 5. Collect data. • Public service providers feedback on projects and services. • Promotes collective between communities and 6. Analyze data. decision making and policy makers may • Manage peoples’ expectations. public service providers 7. Prepare report. feel threatened by the • In cases where social audit teams • Prevents abuse of funds • Enhances citizen participation social audit process. cannot access accurate public 8. Communicate findings. and corruption in monitoring access and records, social audits can focus on user 9. Advocate for service • Helps assess the impact quality of services feedback and advocate for improved improvement and changes. of projects record-keeping over time. 10. Promote feedback and institutionalization of social audit process. 4. A citizen report card is an assessment of public services by the users (citizens) through client feedback surveys. It goes beyond data collection to being an instrument for exacting public accountability through extensive media coverage and civil society advocacy that accompanies the process. • Information collected at the 1. Identify the issues to be • Results are representative • Expensive compared • Develop an understanding of the city, state, or national level covered in the survey. • Obtains credible to other monitoring sociopolitical context in which the via a survey questionnaire of 2. Design the survey instrument. feedback on users’ tools survey is being implemented. individuals or households perceptions regarding • Requires considerable • The organization that conducts the 3. Conduct the survey. • Indicators determined by service delivery human resources as citizen report card should be credible, researchers 4. Analyze and interpret the well as experience with nonpartisan, skilled, experienced, and • Monitors the data. statistical techniques committed to long-term change. • Formal stratified random effectiveness of public sampling used to ensure that 5. Disseminate findings to key spending across areas • Requires specialized • Generate strong media and civil the data is representative of stakeholders. and sectors organizations with society support to publicize the findings, the underlying population 6. Advocate for improvements. • Establishes benchmarks survey skills and engage with citizens, and ultimately • Actual perceptions about to promote performance experience generate a constituency for change in assessment of services are improvements the long run. recorded as an output • Assesses whether • Repeat surveys on a regular basis • Feedback given to service programs are achieving as one report card may result in little providers and the government desired objectives change in service provision and further increase dissatisfaction of users. • Consider hiring local firms and institutes to conduct the surveys for capacity- building purposes. Participatory and Third Party Monitoring in World Bank–Financed Projects  27 (continued) Objectives/Characteristics Steps Benefits/Uses Limits/Challenges Implementation Tips 5. Citizen satisfaction surveys provide a quantitative assessment of government performance and service delivery based on citizens’ experience. Depending on the objective, the surveys can collect data on a variety of topics ranging from perceptions of performance of service delivery and elected officials to desires for new capital 28  How-to Notes projects and services. • Provides feedback on citizens’ 1. Select sample and classify • Findings are • Implementation • Keep the survey short. perceptions of the adequacy users/citizens. representative requires considerable • To measure government responsiveness and efficiency of government 2. Design survey. • Can provide baseline human resources in service delivery, repeat surveys on a services data and key inputs and skills/experience regular basis, as one survey may only 3. Launch survey and data with statistical survey • Monitors citizens’ access to collection. for an evaluation of a provide baseline information. and quality of basic services program or project techniques • Ensure that the agency that conducts 4. Analyze survey results. • Usually only a small • Guides government’s priorities • Repeated surveys can the survey is credible, nonpartisan, in policy planning and service 5. Disseminate survey results. provide an assessment number of people can skilled, and experienced. delivery of the responsiveness of directly participate • Think carefully about the wording and • Assesses community needs government officials • Can be time ordering of questions to avoid bias and • Surveys generally consuming have local staff phrase questions in the contain relatively short • Results often not local language. questionnaires available for a long period of time 6. Public expenditure tracking surveys (PETS) involve citizen groups tracing the flow of public resources for the provision of public goods or services from origin to destination. These surveys can help to detect bottlenecks, inefficiencies, or corruption. • Diagnostic or monitoring tool 1. Identify scope, purpose, and • Strengthen oversight • Government officials • For the results to be credible, the survey to understand problems in actors of PETS. • Identify systemic do not always fully and analysis must be undertaken by budget execution 2. Design the questionnaire and problems in release budget and local, independent researchers. • Data collected from different sample. intergovernmental expenditure data, • Relevant government agencies should levels of government including transfers which makes it more serve on the advisory committee 3. Conduct the survey. difficult to track frontline service delivery units • Uncover overseeing the PETS if there is to be 4. Perform data analysis and expenditures ownership and follow-up on the results. • Heavy reliance on record mismanagement, interpretation • CSOs might have reviews but also interviews inefficiency, or corruption • Collaborate with the media to 5. Disseminate findings to key limited technical disseminate the results. • No standardized instrument; • Generate evidence to stakeholders. capacity to analyze depends on perceived inform ongoing and 6. Encourage institutionalization budget data problems and the nature of future budget debates public resource flows of the initiative. Note: For more information on each tool, please visit the Social Accountability E-Guide (FURL: SAEguide), Social Accountability Database (FURL: SAccdb), and/or Social Development Guidance/How- To Notes. Notes are available at the World Bank’s Social Development website (www.worldbank.org/socialdevelopment).