80707 A Review of Capacity Development Results Measurement in World Bank Projects The Need for a Systematic Approach 1 Copyright © May 2011 The World Bank 1818 H Street, N.W. Washington, D.C. 20433, USA About the World Bank Institute (WBI) WBI’s mission is to be a global facilitator of capacity development for poverty reduction, helping leaders, institutions, and coalitions address their capacity constraints to achieving development results. For more information, visit www.worldbank.org/wbi. About the WBI Capacity Development and Results Practice (WBICR) WBICR provides advisory services, knowledge products, action research and knowledge sharing that inform and support the design and management of results-focused capacity development strategies, policies, and programs. For more information, email capacity4change@worldbank.org. 2 Contents Introduction ................................................................................................... 5 A Systematic Approach to Results Measurement .......................................... 6 Figure 1: CDRF Capacity Development Process Table 1: Institutional Capacities and Their Contributing Characteristics Effective Results Measurement in World Bank Capacity Development Projects .................................................................................. 9 Emerging Patterns for Results Measurement and Indicators in Sectors Table 2: Examples of Assessing Targeted Changes in Stakeholder Ownership Table 3: Examples of Assessing Targeted Changes of Policy Instruments Table 4: Examples of Assessing Targeted Changes in Organizational Arrangements Missed Opportunities for Assessing Capacity Development Results .......... 15 Emphasis on Outputs Emphasis on Intermediate Outcomes Other Potential Problems Table 5: Examples of Mistaking Outputs for Results Table 6: Examples of Mistaking Intermediate Outcomes for Results Conclusions and Recommendations for Strengthening the Measurement of Results .............................................................................. 20 Annex .......................................................................................................... 23 A—Process for Selecting and Reviewing Projects B—List of Projects Reviewed C—Institutional Capacities and Their Contributing Characteristics D—Overview of the Institutional Capacity Indicators Database References ................................................................................................... 36 World Bank Institute Capacity Development and Results Practice May 2011 3 4 Introduction A World Bank Institute (WBI) review of 194 World Bank operational projects with substantial capacity development components revealed critical issues related to the measurement of capacity development results. This exploratory exercise uncovered both potential strengths and shortcomings in current practice and also pointed out ways that results could be improved for future capacity development efforts. Recent studies assert that training and technical assistance activities do not consistently achieve or document intended outcomes. An evaluation conducted by the World Bank Independent Evaluation Group, Capacity Building in Africa, found the Bank “has not developed indicators to define capacity building outcomes” and, by extension, “has not developed a body of knowledge on what tools should be applied and how in different country and sector circumstances” (p. 44, World Bank 2005). Other studies and reviews have drawn similar conclusions, highlighting overarching challenges such as the lack of a conceptual framework and the poor articulation of a results chain (OECD 2005, 2006; World Bank 2006, 2008; Taylor and Clark 2008). Despite donor allocations of more than $30 billion per year on capacity develop- ment activities, a lack of consensus remains on what these activities include and what results should be expected. If programs regularly fail to assess and document capacity impact within their monitoring and evaluation (M&E) systems, development practitioners miss the opportunity for compiling evidence on which interventions are most effective in different situations. WBI developed the Capacity Development and Results Framework (CDRF) in response to this challenge, to provide a systematic guide for results measurement of capacity development interventions. WBI’s Capacity Development and Results practice (WBICR) examined Bank projects to explore the potential value and feasibility of using the CDRF for assessing the results of capacity development interventions. The review process relied on the analysis of project appraisal documents (PADs), which describe how the results of lending operations will be monitored, assessed, and documented throughout a project’s lifetime. This exercise aimed to find examples of indicators for illustrating the application of the CDRF. However, broader lessons emerged related to the state of current results measurement and the need for guidance and improvement. This report shares the main findings of the review, including common pitfalls in measuring results, current practices that are effective, and implications for the way forward. Annex A describes the methodology used and Annex B lists the projects reviewed. 5 A Systematic Approach to Results Measurement WBI developed the CDRF through an ongoing consultative process within the World Bank and among international development practitioners. Over the past three years, outputs of the CDRF have been presented to and reviewed by Bank personnel and client country audiences. Presentations to international audiences include the High Level Retreat on the Effectiveness of International Development Training (Berlin, June 2008) and the Forum on Improving the Results of Learning for Capacity Building (Washington, DC, June 2009). This consultative process allowed the CDRF to build on existing models for capacity assessment and establish its comprehensive and systematic approach for guiding, measuring, and communicating the results of capacity development interventions. The CDRF is a strategic and country-led approach to capacity development that emphasizes the empowerment of local agents through learning, knowledge, and innovation. The CDRF serves as a valuable tool for evaluators and project teams by articulating the results chain of capacity development activities—the progression from needs assessments and interventions to outcomes and impact toward development goals. Figure 1 presents an overview of the capacity development process as defined by the CDRF. Figure 1. CDRF Capacity Development Process Development Goal Development Actions s Sup ion ply s t itutholder • In cre Inst In ke po ase e ituti and hen sta • Im licy in fficie ons Dem ngt p Stre rshi p st nc org rove e rumen y of • ne aniz f t Resources ow atio fective s nal n arra ess of • Financial nge men • Human ts • Technology • Infrastructure Agents of Change • Raised awareness Intermediate Capacity Outcomes • Enhanced skills • Improved consensus and teamwork • Strengthened coalitions • Enhanced networks • Increased implementation Knowledge Services know-how Task teams and other development practitioners can use the CDRF to assess and strengthen the results orientation of capacity development interventions. In particular, the CDRF provides conceptual guidance for a capacity development project, to identify the initial targeted capacity constraints, needed change 6 process(es), and indicators assigned to track this change. This guidance includes adaptable sets of intermediate and final outcome indicators that can be flexibly applied across sectors and countries. This project review focuses on the final outcome level and measuring changes in institutional capacity. The main technical features of the CDRF include a clear specification of institutional capacity factors that can be enhanced through capacity development interventions. Development objectives are defined in terms of improvements in stakeholder ownership, policy-related and organizational capacity areas as follows: Strength of stakeholder ownership The formal and informal political, economic and social forces that determine the priority that government, civil society, and the private sector give to a development goal. Shifts in this area are reflected by widespread changes in attitudes or behaviors that cut across society. Efficiency of policy instruments The administrative rules, laws, regulations, standards, and other formal incentives that a society uses to guide stakeholder actions to achieve development goals. Policy-related capacity changes extend beyond the design or implementation of a new policy, rule, or procedure to focus on the effects on stakeholder actions that have occurred as a result. Effectiveness of organizational arrangements The systems, rules of action, processes, personnel and other resources that state or non-state stakeholders bring together within a specific organization to achieve a development goal. These capacity changes extend beyond the development of individuals’ knowledge and skills to focus on the results that occur due to shifts in organizational behavior and performance. For each of these institutional capacity areas the CDRF specifies contributing characteristics that can be enhanced through capacity development interventions (table 1, next page). Annex C provides an in-depth description of the 19 characteristics. The CDRF’s list of capacity characteristics provides a detailed guide to identify local institutional capacity constraints to the achievement of a given development goal. This process informs the customized design of capacity development interventions. In addition, this set of capacity characteristics provides a comprehensive and consistent approach for the measurement of capacity development results. Assessing results across interventions and sectors provides comparable data that, over time, could contribute to a body of knowledge about what does and does not work in capacity development based on practical experiences. 7 Table 1. Institutional Capacities and Their Contributing Characteristics Characteristics Characteristics Characteristics Contributing to the Contributing to the Contributing to the Strength of Stakeholder Efficiency of Policy Effectiveness of Ownership Instruments Organizational Arrangements • Commitment of social • Clarity in defining rights • Clarity of mission and political leaders and responsibilities • Achievement of • Compatibility of social • Consistency outcomes norms and values • Legitimacy • Operational efficiency • Stakeholder • Incentives for compliance • Financial viability participation in and probity setting priorities • Ease of administration • Communications and • Stakeholder demand • Risk for negative stakeholder relations for accountability externalities • Adaptability • Transparency of • Suitable flexibility information to • Resistance to corruption stakeholders This review of Bank projects indicates how the CDRF can be used to effectively assess capacity development needs and results. The exercise revealed cross- cutting examples of how the CDRF’s capacity characteristics can feasibly be measured in multiple sectors. At the same time, the review identified many cases where project teams did not have sufficient guidance to measure their targeted capacity changes. These findings illustrate how the CDRF can improve M&E practices for capacity development interventions by facilitating a consistent, detailed approach for defining, monitoring, assessing, and communicating results. The evidence suggests that shortcomings related to the measurement of capacity development results can be mitigated by integrating use of the CDRF into Bank operations. 8 Effective Results Measurement in World Bank Capacity Development Projects WBICR examined PADs for projects with substantial capacity development components to identify indicators and measures for assessing each of the CDRF’s capacity characteristics. The good practice examples that emerged confirmed that the use of high-quality, precise results measures is feasible across sectors and types of interventions. The presence of these examples, coupled with the missed opportunities described in the next section, illustrate how the CDRF’s detailed results nomenclature and assessment process can facilitate greater accuracy, consistency, and comprehensiveness in the measurement of capacity develop- ment results. Measuring widespread beliefs, attitudes, and behaviors can capture broad changes in the strength of stakeholder ownership of capacity development. Project teams could assess such changes by examining how aware stakeholders are of the gov- ernment’s actions related to a particular development goal; how much stakehold- ers participate in decision-making; and to what degree stakeholders hold public service providers accountable for achieving the development goal. Assessment methods could also focus on the demonstrated commitment of leaders to the development goal as evidenced through votes, public speeches or actions, and budget allocations. Alternatively, the compatibility of current social norms with the development goal could be measured through household surveys and other methods to examine beliefs and behaviors among various population groups. Some Bank projects accurately targeted, monitored, and assessed changes in capacity related to stakeholder ownership. For example, in sub-Saharan Africa, efforts in one country to increase the capacity of health facilities and programs for treating and/or preventing HIV/AIDS transmission have been thwarted by widespread prejudice against infected individuals and a pervasive lack of understanding about transmission. A project component focused on media broadcasting and community education to reduce this stigma and change behaviors so that treatment and prevention programs would be supported and sought out. Surveys assessed whether targeted changes in attitudes and behaviors occurred. Table 2 provides examples of similar measures to assess shifts in stakeholder ownership across sectors. Assessments of the efficiency of policy instruments, according to the CDRF, focus on measuring how administrative rules, regulations, procedures, or other formal mechanisms guide stakeholder actions to achieve development goals. Useful indicators might assess how clearly the regulation or rule clarifies the rights and responsibilities of stakeholders or to what degree the formal mechanism is viewed as fair and appropriate by the local population. Other considerations include the degree of burden that the rule or process places on the implementers and the 9 Table 2. Examples of Assessing Targeted Changes in Stakeholder Ownership Development Indicator Related Measure Data Goal to Capacity Need Source Promote Perception among • Percentage of surveyed residents Intercept environ- residents that who perceive that walking and surveys mentally Bus Rapid Transit cycling have become safer and sustainable System can be more comfortable in project area urban safely accessed by • Proportion of Bus Rapid Transit transport bicycle or on foot System riders accessing the system through bicycles or on foot Establish a Transparency Percentage of households that Institutional functioning of information reported hearing about govern- Reform and local regarding ment efforts from an official source: Capacity government decentralization How do you hear about what the Building system efforts government is doing? Project • No source National • Relatives, friends, neighbors, Public co-workers Services • Community bulletin board Survey • Village headman/headwoman • Paramount or section chief/ chiefdom officials • Newspaper • Radio • TV • Other Improve Participation • Participation rate of poorest and Manage- public of community vulnerable community members ment services in stakeholders in planning and decision-making Information targeted in decisions meetings System of urban areas regarding local Oversight • Participation rate of women in public services Consultant planning and decision-making meetings • Percentage of kelurahans (urban wards) with established community boards of trustees Examples from World Bank lending operations incentives or barriers faced by those who should comply. Whether the mechanism is creating unintended outcomes is an important consideration in some contexts. Also, project teams could assess resistance of a new rule or procedure to corruption by examining the degree to which actions are removed from individual decision-making. Examples of effective results measurement related to the efficiency of policy instru-ments extend beyond the establishment or implementation of a new formal mechanism (such as a policy, strategy, or procedure) to examine what happened as a result. In southeastern Europe, for example, border crossing 10 Table 3. Examples of Assessing Targeted Changes in Policy Instruments Development Indicator Related Measure Data Goal to Capacity Need Source Increase Compliance with • Percentage of public school Ministry of the quality credentialing teachers who meet professional education of primary requirements standards for licensing teacher education among teachers licensing data Improve Clarity regarding Clear designation for policies and Records the health oversight responsibilities on professional of Central status of the responsibilities for accreditation, certification, Project population different types of and school licensure for each Coordi- health professional profession. Is a body established to nation Unit education provide oversight on standards for programs accreditation, content, and conduct [yes/no] for medical, dental, nursing and midwifery education. Provide Use of automated • Percentage of field audits Tax citizens with selection selected by automated committee better public procedures for procedure data services and field audits infrastructure Examples from World Bank lending operations police and customs officials were known to individually decide which vehicles would be subjected to manual searches and requests for informal payments. A new automated risk management procedure was implemented to dictate which vehicles would be manually examined based on various predetermined profiling characteristics. The project team tracked changes in the officials’ corrupt behavior by measuring the percentage of customs transactions that adhered to new investigative guidelines and the occurrence of irregularities per number of examinations (targeted to increase as officials become more selective in conducting searches). Table 3 provides additional examples of how the efficiency of policy instruments can be assessed across sectors. The CDRF outlines how the effectiveness of organizational arrangements can be evaluated by focusing on how an organization performs related to the achievement of a particular development goal. Characteristics that might be assessed include the level of organizational efficiency for producing outputs related to the development goal (quality and cost) or the achievement of outcomes contributing to the development goal. The financial viability or probity of the organization can also be measured, as well as its ability to adapt in changing circumstances or foster innovation. Stakeholder perceptions can also be explored: How well do they understand the overall mission of the organization related to the development goal and how effectively are they communicated with and engaged to support the organization’s work. 11 Table 4. Examples of Assessing Targeted Changes in Organizational Arrangements Development Indicator Related Measure Data Source Goal to Capacity Need Improve Financial viability Operational cost ratio Audited financial sustainable of the Water (percentage cost recovery) of statements access to safe and Sewerage Water and Sewerage Authority of Water and water supply Authority Sewerage Authority Improve Level of employ- Percentage of respondents, Living Standards employment ment of individuals ages 14 and older, who Measurement rates using labor office indicated they found their job Survey services through the labor office Improve Cost of land regis- Per unit cost and time of Intendance land tenure tration process at regularization process titling infor- security test department mation system Examples from World Bank lending operations The arrangements of specific organizations are often assessed in the results frameworks of Bank projects. In Southeast Asia, for example, a centralized electricity company is highly subsidized to address a country’s energy needs. Capacity development activities focus on reforming the organizational arrangements to promote long-term sustainability through cost recovery. The financial viability of the company can then be monitored through measurements of the self-financing ratio, the rate of return on revalued assets, and the debt service coverage. Table 4 provides examples of other approaches in Bank projects for assessing the effectiveness of organizational arrangements. The presence of these good measures for assessing the CDRF’s institutional capacity characteristics helps to confirm that the use of a more precise and comprehensive results measurement system is feasible within the competing demands of Bank operations. The ongoing collection of effective examples can serve as a resource for project teams in developing appropriate adapted results measures for specific capacity characteristics, particularly as issues of comparability across sector-specific practices and requirements become better understood. As a result of this review, WBICR established the Institutional Capacity Indicators Database to provide a searchable catalogue of indicators for monitoring and measuring changes in capacity (see annex D). Emerging Patterns for Results Measurement and Indicators in Sectors Variations in assessment practices by sector provide concrete examples of why an adaptable and comprehensive approach is needed for assessing capacity development results. If models for effective measurement in a specific sector are understood within a detailed common framework (such as the CDRF), then good practices can be more readily communicated and adapted for use in other sectors. 12 Furthermore, common sector-specific omissions in results measurement are more likely to be avoided if capacity needs assessments and results frameworks are designed in terms of a standard list of measurable capacity characteristics. In the projects analyzed, certain capacity challenges seemed more likely to be targeted and monitored in some sectors than in others. Systematically exploring these sector-specific approaches to assessing capacity development results could reveal effective patterns for best practice sector models. The CDRF can guide and facilitate the compilation of such models across sectors, capacity characteristics, and types of interventions to support a systematic and comprehensive approach to assessing capacity development results. In this sample, assessments of organizational arrangements were most readily available in sectors within the Sustainable Development and Human Development Networks. For example, project results frameworks included indicators related to the outcomes produced by a health facility, the financial viability of a water company, or the satisfaction of users of an agriculture advisory service. In contrast, indicators related to the efficiency of policy instruments were more likely in sectors within the Poverty Reduction and Economic Management Network. For example, these operations included indicators related to the level of resistance of the public budgeting process to exceptional spending (how resistant the process is to corruption); the accuracy with which ministry personnel understand their responsibilities related to procurement rules (clarity of the new rules in defining responsibilities of stakeholders); or the degree to which taxpayers adhere to tax requirements (incentives for compliance with tax law). Projects missed key capacity areas that should have been assessed and tracked. In cases where teams isolated precise capacity characteristics, the absence of an overarching framework for capacity assessment could mean that they overlooked other types of relevant capacity characteristics. For example, in the Europe and Central Asia region, a project team designed a lending operation to improve the financial viability of a state-owned company responsible for supplying heat in a capital city. They assigned indicators to evaluate the company’s progress in pass¬ing the costs of supplying heat onto final consumers. These measures focused solely on organizational performance and failed to monitor the broader and more complex capacity development story. For instance, heating tariffs were not set at the cost recovery level. Members of the municipal councils, who were in charge of regulating the heating tariffs, resisted adjusting the tariffs because it would have been politically unpopular. Thus, key constraints were overlooked that needed to be addressed to achieve the company’s financial viability related to stakeholder ownership (the commitment of leaders and general attitudes within the population) and the policy arena (suboptimal heating tariffs). 13 The CDRF can help to strengthen M&E practice even for those projects with effective indicators and measures in place. The use of the CDRF to guide a systematic approach would allow practitioners across sectors to identify and share promising models of capacity development results measurement, more comprehensively assess capacity needs within a broader context, and identify and resolve short-comings or omissions in results frameworks. 14 Missed Opportunities for Assessing Capacity Development Results Aside from identifying examples of good practice, the review also examined com- mon pitfalls related to the measurement of capacity development results in Bank projects to clarify how and why the CDRF’s systematic approach is needed. World Bank projects are frequently designed to address “limited institutional capacity,” but this term is not clearly defined or consistently understood. Common references to individual, organizational, institutional or societal capacity do not necessarily translate to clear measurable outcomes. Furthermore, generic guidance for good M&E prac- tice has proven to be insufficient for identifying and assessing the change processes and logical sequencing needed for achieving targeted capacity development results.. Emphasis on Outputs Counting the tangible products and services provided by Bank operational projects does not adequately describe which changes in institutional capacity occurred as a result of the interventions. The training of individuals or groups must result in changed behavior, such as the use and application of enhanced knowledge or skills, improved collaboration or teamwork, or the design and implementation of a new strategy or product. Even these changes, which certainly can be measureable outcomes, are only an intermediate step along the results chain. Projects often confuse outputs as outcomes and present them as intermediate or final results. For example, an urban infrastructure and service delivery project included an institutional strengthening component that supported training on financial management and preparing urban development plans. A results indicator for this component listed in the PAD is the number of trainees trained. Similarly, a project aimed at developing institutional capacity for increasing access to basic urban services identified the number of bus/lorry stands constructed as a key inter- mediate outcome measure. Table 5 presents other examples of output measures appearing as inappropriate measures for intermediate outcomes or final results. The outputs of project activities cannot serve as effective proxy measures for results. For example, projects often track measures such as the number of people trained or the number of positions filled. Quantifying these outputs could serve an important monitoring function for assessing whether a project is being implemented as planned. Such measures should be appropriately assigned for early steps in a project’s results chain to better ensure accountability. Emphasis on Intermediate Outcomes Assessing whether project activities achieve their short-term objectives is a critical step in monitoring progress along a results chain, but this does not provide insight on program impact. Capacity development efforts often include training or techni- cal assistance activities with learning objectives related to enhanced knowledge 15 Table 5. Examples of Mistaking Outputs for Results Targeted Project Assigned Outcome Measures Shortcomings of Measures Outcomes Improved • Health personnel receiving The provision of training and performance training, such as the number hiring for positions does not of the health of nurses, anesthetists, and automatically result in improved workforce clinical officers performance. No measures for performance are included (such • Approved positions filled as quality, efficiency, and probity). by qualified health workers Also, “qualified” is not clearly (percentage) defined. Improved • Number of company Training focused on the commercial employees receiving training company’s management management to use the Commercial systems does not guarantee of the National Management System, better management. Outcome Electric Power the Corporate Resources measures should focus on Company Management Information individual and organizational System, and the Incidence performance changes. Recording Management System (number per year for each management system) Improved living • Number of sewer This measure is poorly defined, conditions of connections benefiting from with “benefiting” and “rehabilita- populations in rehabilitation works tion works” not specified. More downstream importantly, the targeted result riverside of improved living conditions is communities not confirmed. Relevant outcome measures should focus on access to a clean water supply, as in the share of water samples meeting defined quality standards. Increased • Cumulative number No assessment is included economic of project sites with regarding the quality of the benefit to local conservation work conservation plans themselves. It communities completed in line with is unclear how completing these through approved conservation plans conservation works will guarantee increased increased economic benefit. • Cumulative number of tourism and Similarly, training youths might youths trained for traditional better protected or might not result in changed arts and techniques cultural and behavior that then might yield natural heritages economic benefit. Increased • Number of new transport No assessment is noted regarding efficiency of the analyst functions filled who will fill these new functions national transport and how their performance agency will contribute to operational efficiency. No measurement is proposed related to quality or cost, that is, aspects of efficiency. Examples from project results frameworks 16 and skills, improved processes (such as consensus or teamwork), and new products (such as action plans, policies or strategies). These changes in status, behavior, relationships and products are also known as intermediate outcomes, which serve as stepping stones to changes in institutional capacity areas—shifts in the strength of local stakeholder ownership, efficiency of policy-related instruments, or effective- ness of organizational arrangements. However, they do not inform on the impact of capacity development interventions. Some Bank projects rely on intermediate outcome measures to provide evidence of capacity development results. For example, a project to strengthen the quality of education for health professionals uses the establishment of an independent national accreditation agency as a key measure to assess results. However, the establishment of this entity represents an altered process for accreditation rather than an improvement in educational quality. This measure does not offer information related to changes in the quality of education. Similarly, a project to increase the access of underserved rural communities to basic health services lists the development of a national health strategy to reduce inequali- ties in access to health services as a project outcome measure. No assessment is included or planned for whether such a strategy then leads to changes in social behavior, policies, or organizational performance that would result in increased access. Table 6 presents additional examples of how tracking intermediate outcomes does not provide insight on project impact. Intermediate outcomes cannot serve as proxy measures for changes in capacity. The confusion between intermediate outcomes and final results that surfaces in project results frameworks underscores the potential value of the CDRF. The CDRF can be used to elucidate a capacity development intervention’s results chain and then guide the assignment of measures for short-term output and intermediate outcome levels. Enhancing the skills of an agency’s workforce, developing a new committee, or formu- lating an action plan could be necessary outcomes on the way to achieving capacity development results, but these alone are insufficient to claim changes in organiza- tional performance or stakeholder actions for achieving a development goal. Other Potential Problems Omissions in measurement distort our understanding about what results are achieved. The misplaced emphasis on outputs or intermediate outcomes was the most frequent problem encountered in project documents, but additional shortcomings further undermine the assessment of interventions. These shortcomings include the failure to evaluate progress in addressing identified capacity constraints, claiming results at the development goal level, and regularly overlooking certain characteristics of capacity. Capacity challenges highlighted in a project’s narrative are sometimes missing from the results framework. For example, the PAD for a municipal development project details the need to increase the efficiency in procedures for urban planning. 17 Table 6. Examples of Mistaking Intermediate Outcomes for Results Targeted Project Assigned Outcome Measures Shortcomings of Measures Outcomes Enhanced • Adoption of the local The adoption of a strategy or capacity of the economic development plan is a learning product— city government strategy by 2009, and budget the application or transfer in promoting allocated to undertake main of knowledge—and does local economic actions from the strategy not reflect changes in the development starting in 2010 performance of the city government. • Implementation of a competitiveness enhancement plan for one economic cluster Increased use of • Percentage of execution of The implementation of an HIV/AIDS and STD Eligible Expenditure Programs action plan or program prevention, care, is a learning product. and treatment No evidence is provided services by groups whether the use of services most at risk changes as a result. Increased • The number of partnerships The formation of collaborative among public research partnerships reflects a development system, private sector and changed process (teamwork) and application other stakeholders and does not necessarily of agricultural lead to the increased innovations among development and application public research of agricultural innovations. organizations, The term partnership is not farmers, private precisely defined. sector and other stakeholders Strengthened • Completion of the [City] The completion of a institutional capacity Transport Master Plan plan does not reflect of transport strengthened institutional authorities capacity. Instead, information is needed on the results that occur because of this new plan. Strengthened health • Development of a The development of the system comprehensive health care strategy itself is a learning finance strategy product. No evidence is collected through this measure for whether more efficient or equitable financing occurs as a result. Improved • Successful implementation No definition for successful environmental of the Environmental implementation is specified. sustainability Management Plan The implementation of a through mitigation plan or strategy does not of impacts guarantee the achievement associated with dam of capacity development results. Examples from project results frameworks 18 Outcomes mentioned in the narrative for the relevant project component include reduction in the number of steps and files required for urban plan preparation and development approvals. However, no measures related to decreasing this adminis- trative burden are included in the project’s results framework. Projects typically do not assess the unintended outcomes of policy changes, even when sector specialists understand that such externalities are likely to occur. For example, changes in transport policy have been associated with increases in STDs because truckers and other workers travel farther from home. Since transport has become recognized as a social vector in the transmission of HIV/AIDS, a project focused on the development and implementation of a country’s strategy to improve transport infrastructure includes activities to educate truckers about high-risk sexual behavior. This education campaign is part of the project’s activities, but no indica- tors are included in the results framework to assess specific learning outcomes or to evaluate whether unintended negative effects of the new infrastructure occurred. Changes in targeted capacity characteristics sometimes go unmeasured despite documented progress toward the achievement of a development goal. In these cases, improvements in development goal-level indicators are reported as evidence of project-level capacity development results. For example, in a project for increas- ing access to secondary education, the project tracked increases in gross enroll- ment rates in lower and upper secondary education and used them as out-come measures to show success. However, no information is included about what changes in capacity have occurred to promote and sustain increased enrollment. Have social norms changed so that more parents are pursuing schooling for their children? Or, perhaps a policy change made it easier for students to enroll? Or, at the organiza- tional level, perhaps local schools are operating more effectively in attracting and retaining the local school-age population? Without charting progress in overcoming a specific capacity challenge, a project team cannot accurately assess whether capacity development activities are making the intended contribution to the development goal or monitor project implemen- tation and make adjustments as needed. This latter aspect is important given the complexity of institutional change processes In the results frameworks examined, Bank operations overlooked certain types of capacity changes. Project teams might identify one or more challenges to address through capacity development activities, but they do not typically draw from a stan- dardized menu of capacity areas and characteristics to help them think through all of the variables constraining local capacity. In particular, this review identified few exam- ples where teams assess characteristics related to the efficiency of policy instruments, such as rules, regulations, standards and formal procedures. Such characteristics in- clude the level of administrative burden imposed or the unintended consequences caused by a formal mechanism. Another aspect rarely evaluated is whether a policy- related instrument is sufficiently flexible to adapt to changing local conditions. 19 Conclusions and Recommendations for Strengthening the Measurement of Results The review of appraisal documents for 194 projects in the Bank’s lending portfolio provided important insights and guidance to strengthen the results orientation of capacity development interventions. First, existing examples of effective results measurement for capacity development interventions illustrate the potential value of the CDRF for more systematically and consistently defining and measuring targeted changes in capacity across sectors. Second, although good M&E practice should consistently include an articulated results chain, the sequence from inputs to results often remains undefined for the capacity development components of projects. The CDRF is designed to demystify capacity development—to spell out how progress can be measured along the path from inputs to results. It is critical to improve the measurement of capacity development results. This exercise highlighted common problems in current practice, including confusing outputs or intermediate outcomes with results and overlooking key capacity chal- lenges in assessing progress toward achieving change. The absence of a detailed, effective, and widely accepted approach for capacity development assessment has important implications for Bank operations: • Uneven quality among results measures produces inconsistencies, even within a single project, in how outcomes are characterized and measured. Teams assign effective measures at times but miss opportunities in other cases for tracking results. This dynamic underscores the need for a more systematic approach. • Lack of a common language in measuring capacity development results makes it difficult to compare outcomes across types of interventions and sectors. If practitioners across projects and sectors identify and characterize capacity changes using a standardized framework, they could accumulate a body of knowledge around which interventions are effective and in what context. Recommendations Integrate the use of the CDRF into Bank projects The 194 project results frameworks collectively illustrate why a more consistent and accurate approach is needed to assess the results of capacity development components. Identifying example measures for evaluating changes related to the CDRF’s capacity characteristics demonstrated the feasibility of using the framework to guide a more consistent and precise assessment process across interventions and sectors. Use of the CDRF among Bank teams and stakeholders is likely to be facilitated by: • Providing tools and resources to support operational staff in using the CDRF— A learning process is required to become familiar with and successfully apply the CDRF. Resources to support this process include the Institutional Capacity 20 Indicators Database and an institutional diagnostic toolkit. Other resources should be explored for guiding practitioners to assess capacity constraints and targeted capacity changes. • Identifying a cadre of champions across sectors to support teams in applying the CDRF—Understanding the institutional capacity areas and their contributing capacity characteristics requires hands-on experience in articulating the results chains for capacity development interventions and defining indicators and measures. A roster of CDRF specialists would help guide and refine this process. Continue the investigation of results measurement for capacity development interventions This review has, at minimum, highlighted recurring shortcomings related to the as- sessment and documentation of outcomes. Understanding current practice better and planning improvements could include: • Examining and revising the policies and procedures guiding results measurement—This review suggests that the current requirements for assessing project results are insufficient to ensure sound M&E practice. For example, the current format for PADs in general and results frameworks in particular does not necessarily prompt teams to define steps along a results chain (that is, outputs versus outcomes). In addition, the review process for required documentation is not effectively preventing common shortcomings in results measurement. • Conducting a scientific study of current practices using a representative sample of Bank projects—This review facilitates understanding the landscape for capacity development results measurement at the Bank. However, more evidence and guidance regarding current practice would be gained by designing a study based on a random sample of projects wherein the findings could be generalized to the larger population of projects in Bank operations. This study could serve as part of a larger study to examine capacity development results and pinpoint best practices. Require practitioners to more systematically monitor and assess results and then use the lessons learned for continual program improvement Strengthening results measurement in the Bank must go beyond refining the project results frameworks in PADs to include iteratively applying the information collected during program design and implementation. The CDRF provides a practical resource for guiding good M&E practice throughout the capacity development project cycle to better ensure that desired capacity changes are effectively targeted and achieved. 21 1 22 ANNEX A Process for Selecting and Reviewing Projects WBICR designed the review of Bank operational projects to locate examples of how the CDRF’s institutional capacity characteristics are assessed in current prac- tice. The project selection and review process evolved as needed to identify projects with substantial capacity development components in a range of sectors. This exercise produced a database of example indicators aligned with the CDRF. The lessons learned regarding the strengths and weaknesses of current practice among Bank teams and their implications for planning improvements are the basis for this report. Project Selection To facilitate the collection of diverse examples for measuring capacity develop- ment results, the sampling process identified projects for review based on their focus in priority sectors, the strength of their capacity development component(s), and the date they became active. • Sectors—Although indicators aligned with the CDRF can be assigned in projects in any sector, the project sampling concentrated on sectors considered to be current strategic priorities for WBI. These included areas within three networks: • Human development—education; health, nutrition and population; HIV/ AIDS; social protection; and children and youth. • Sustainable development—agriculture and rural development; social development; infrastructure; energy; environment; urban development; and water. • Poverty reduction and economic management—economic policy and debt; gender and development; poverty reduction; public sector governance; and trade. • Capacity development—The selection process targeted projects in which training and technical assistance activities were incorporated into components for developing local capacity. The online search function for locating Bank projects (available at www.worldbank.org/projects) is effective for locating such projects only if “capacity development” or other useful identifier is entered as keywords. Each project synopsis posted online was examined and then project descriptions within PADs were scanned. WBICR selected those Bank operations that emphasized capacity development for a full review. • Project age—Through the implementation of and adherence to the Bank’s results agenda, teams progressively strengthened the M&E frameworks of individual projects. To capitalize on this increasing results-based focus, the sampling process focused mainly on projects that became active in 2006 or later. WBICR determined this cutoff because project results frameworks followed a new format after the mainstreaming of the results-based country assistance strategy in the second half of fiscal year 2005. 23 WBICR also identified and reviewed projects based on recommendations from different team members and other sources, such as the lists of capacity devel- opment projects spotlighted by the World Bank Operations Policy and Country Services unit. Therefore, most but not all of the projects fit within the categories outlined above. Document Review Once WBICR identified projects as having a capacity development component, they examined the PAD for model examples to illustrate how the CDRF’s institu- tional capacity characteristics are assessed. Three sections of the PAD were explored to map indicators and measures to the CDRF: • Results framework—The project results framework and monitoring arrangements table for a Bank project are typically organized by project component. Project teams assign indicators for results, intermediate outcomes, and outputs levels, but this hierarchy is not consistently used. Thus, the review considered any indicators assessing capacity development placed anywhere in the results framework. • Project description—The assessment of capacity development results is concerned with the outcomes achieved by capacity development interventions. In some cases, it is unclear whether the project components listed in a results framework involve training and technical assistance activities. In these cases, the review included the more detailed review of the narrative sections describing project activities. The annex following the results framework in the PAD often served as a source for this information. • Country and sector background—In some cases, the context for indicators and measures listed in the project results framework is unclear. Which capacity constraints are the project designed to address? Are the proposed measures appropriate for assessing progress in overcoming these capacity constraints? At times, the selection of example indicators and measures required a review of additional narrative sections of the PAD to better understand the project context. This information was usually available in either the opening section (Strategic Context and Rationale) or in the first annex (Country and Sector or Program Background). WBICR identified indicators and measures through this process to illustrate the use of the CDRF if they met each of conditions below. • They effectively measured an institutional capacity area. Specifically, the outcome indicator reflected changes in one of three areas: • Widespread perceptions or behaviors that extend beyond specific organizations—possibly measuring the strength of stakeholder ownership related to a particular development goal • Aspects related to administrative rules, laws, regulations, or standards to ensure the coordinated and efficient pursuit of a development goal by diverse stakeholders—possibly measuring the efficiency of policy instruments 24 • The use of systems, rules of action, processes, personnel, and other resources within specific organizations to advance the development goal— possibly measuring the effectiveness of organizational arrangements. • They were associated with a development goal. The outcome indicator must reflect progress related to a specified economic or social benefit. • They focused on a specific characteristic of institutional capacity. Indicators that cannot be disaggregated to the level of a specific capacity characteristic do not serve as strong examples. In limited cases, an indicator might serve as a proxy measure as long as some part of it represents a specific capacity characteristic. • They adhered to good M&E practice. This exercise aimed to find existing good examples that did not need revision to be effective. To the extent possible, each indicator and measured adhered to the SMART criteria (specific, measurable, achievable, relevant, and time-bound). Indicators that met these conditions became examples for inclusion in the Insti- tutional Capacity Indicators Database (see annex D). During this process, WBICR noted recurring issues related to the measurement of capacity development results and began recording these observations and collecting examples. These observa- tions formed the basis for this report. 25 ANNEX B List of Projects Reviewed To look up these projects, visit www.worldbank.org and search under Projects Portfolio using the appropriate keywords. The Projects Portfolio contains data on World Bank lending projects from 1947 to the present. 1. Afghanistan - Pension Administration and Safety Net Project 2. Afghanistan - Second Customs Reform and Trade Facilitation Project 3. Angola - Municipal Health Service Strengthening Project 4. Angola - Water Sector Institutional Development 5. Argentina - Metropolitan Areas Urban Transport Project 6. Argentina - Road Safety 7. Azerbaijan - Rail Trade and Transport Facilitation 8. Bangladesh - Chittagong Water Supply Improvement and Sanitation Project 9. Bangladesh - Clean Air and Sustainable Environment Project 10. Bangladesh - Empowerment and Livelihood Improvement “Nuton Jibon” Project 11. Barbados - Second HIV/AIDS Project 12. Benin - Health System Performance Project 13. Benin - Increased Access to Modern Energy Project 14. Bhutan - Second Urban Development Project 15. Bosnia and Herzegovina: Social Safety Nets and Employment Support Project 16. Bosnia and Herzegovina: Sarajevo Waste Water Project 17. Botswana: Integrated Transport Project 18. Botswana: Morupule B Generation and Transmission Project 19. Botswana: Northern Botswana Human Wildlife Coexistence Project 20. Brazil - AIDS-SUS (National AIDS Program - National Health Service) Project 21. Brazil - Fiscal Consolidation for Efficiency and Growth Development Policy Loan Program 22. Brazil - Municipal APL: Sao Luis Enhancing Municipal Governance and Quality of Life Project 23. Brazil - Pernambuco Sustainable Water Project 24. Brazil - Sao Paulo Sustainable Rural Development and Access to Markets Project 25. Brazil - Ceara Regional Economic Development: Cidades do Ceara 26. Bulgaria - Municipal Infrastructure Development 27. Bulgaria - Social Inclusion Project 28. Burkina Faso - Community-Based Rural Development Project 29. Burkina Faso - Energy Access Project 30. Burkina Faso - Post-Primary Education Project 31. Burkina Faso - Second Community Based Rural Development Project in support of the National Program for Decentralized Rural Development (APL) 32. Burkina Faso and Ukraine - Multi-Country statistical Capacity Building Program (STATCAP) First Phase Project 33. Burundi - Financial and Private Sector Development Project 34. Burundi - Public Works and Urban Management Project 26 35. Cambodia - Higher Education Quality and Capacity Improvement Project 36. Cambodia - Public Financial Management and Accountability Project 37. Cambodia - Demand for Good Governance Project 38. Cameroon - Transparency and Accountability Capacity Development Project 39. Cape Verde - Small and Medium Enterprises (SME) Capacity Building and Economic Governance Project 40. Chad - Public Financial Management Capacity Building Project 41. China - Anhui Medium Cities Urban Transport Project 42. China - Bengbu Integrated Environment Improvement Project 43. China - Chongqing Urban Rural Integration Project 44. China - Guizhou Cultural and Natural Heritage Protection and Development Project 45. China - Jilin Agricultural Product Quality and Safety Project 46. China - Ningbo New Countryside Development Project 47. China - Wuhan Second Urban Transport Project 48. China - Sustainable Development in Poor Rural Areas 49. Colombia - Consolidation of National Public Management Information Systems Project 50. Colombia - Mainstreaming Sustainable Cattle Ranching Project 51. Colombia - Strengthening Public Information, Monitoring and Evaluation for Results Management Project 52. Congo, Democratic Republic of - Growth with Governance in the Mineral Sector Project 53. Congo, Democratic Republic of - Street Children Project 54. Dominican Republic - Municipal Development Project 55. Dominican Republic - Second Phase Adaptable Program Loan for the Health Sector Reform Project 56. Dominican Republic - Water and Sanitation in Tourist Areas 57. East Africa Public Health Laboratory Networking Project 58. Egypt, Arab Republic of - Wind Power Development Project 59. Egypt, Arab Republic of - Second Pollution Abatement Project 60. El Salvador - Local Government Strengthening Project 61. Ethiopia - Road Sector Development Program and Fourth Adaptable Program Loan Project 62. Ethiopia - Rural Capacity Building Project 63. Ethiopia - Urban Local Government Development Project 64. Georgia - Avian Influenza Control and Human Pandemic Preparedness and Response (AIHP) (APL) Project 65. Georgia - Kakheti Regional Roads Improvement Project 66. Georgia - Regional and Municipal Infrastructure Development Project 67. Georgia - Third East West Highway Improvement Project 68. Ghana - Nutrition and Malaria Control for Child Survival Project 69. Ghana - Second Health Program Support Project 70. Ghana - Social Opportunities Project 71. Ghana - Third Natural Resources and Environmental Governance Development Policy Operation Project 72. Guatemala - Expanding Opportunities for Vulnerable Groups Project 27 73. Guatemala - Maternal-Infant Health and Nutrition Project 74. Guinea-Bissau - Second Economic Governance Reform Grant Program 75. Haiti - Urban Community Driven Development Project (PRODEPUR) 76. Honduras - Power Sector Efficiency Enhancement Project (PROMEF) 77. Honduras - Community Based Education Project 78. India - Capacity Building for Industrial Pollution Management Project 79. India - Elementary Education Project 80. India - First Phase of the National Cyclone Risk Mitigation Project 81. India - Haryana Power System Improvement Project 82. India – National Agricultural Innovation Project 83. India - Orissa Rural Livelihoods Project 84. India - Second Madhya Pradesh District Poverty Initiatives Project (MP-DPIP II) 85. India - Sustainable Urban Transport Project 86. India - The Financing Energy Efficiency at MSMEs Project 87. India - Andhra Pradesh Municipal Development Project 88. Indonesia - Health Professional Education Quality Project 89. Indonesia - Power Transmission Development Project 90. Indonesia - Third National Program for Community Empowerment in Urban Areas Project 91. Jamaica - Education Transformation Capacity Building Project 92. Jamaica - Rural Economic Development Initiative Project 93. Kazakhstan - Tax Administration Reform Project 94. Kenya - Agricultural Productivity and Agribusiness Project 95. Kenya – Electricity Expansion Project 96. Kenya - Youth Empowerment Project 97. Kosovo - Institutional Development for Education Project 98. Kosovo - Public Sector Modernization Project 99. Kosovo - Real Estate Cadastre and Registration Project 100. Kyrgyz - Water Management Improvement Project 101. Kyrgyz Republic - Second On-Farm Irrigation Project 102. Lao - Second Phase of the Rural Electrification Project 103. Lao People’s Democratic Republic - Technical Assistance for Capacity Building in the Hydropower and Mining Sectors Project 104. Lesotho - Second Phase Health Sector Reform Project 105. Lesotho - Second Phase of Water Sector Improvement Project 106. Macedonia - Municipal Services Improvement Project 107. Macedonia - Trade and Transport Facilitation in Southeast Europe Project 108. Malawi - Multi-Sectoral AIDS Project 109. Maldives - Third Education and Training Project 110. Mali - Fostering Agricultural Productivity Project 111. Mauritius - Infrastructure Project 112. Mexico - Social Protection System in Health Project 113. Mexico - Sustainable Transport and Air Quality Project 114. Mexico - Urban Transport Transformation Project 115. Moldova - Road Sector Program Support Project 28 116. Moldova - Second Poverty Reduction Support Credit 117. Moldova - Second Rural Investment and Services Project - Second Phase of the Rural Investment Services Program 118. Moldova - Health Services & Social Assistance 119. Mongolia - Multi-Sectoral Technical Assistance Project 120. Morocco - MA-Regional Potable Water Supply Systems Project 121. Mozambique - National Decentralized Planning and Finance Project 122. Mozambique - Second Adaptable Program Loan for the Energy Development and Access Project 123. Mozambique - Higher Education Science and Technology 124. Mozambique - ProMaputo, Maputo Municipal Development Program 125. Nepal - Second Health Nutrition and Population (HNP) and HIV/AIDS Project 126. Nicaragua - Education Project (PASEN) 127. Nicaragua - Land Administration Project (PRODEP) 128. Niger - Agro-Pastoral Export and Market Development Project 129. Niger - Local Urban Infrastructure Development Project 130. Niger - Reform Management and Technical Assistance Project 131. Nigeria - Community and Social Development Project 132. Nigeria - Second HIV/AIDS Program Development Project 133. Nigeria - State Education Sector Project 134. Pakistan - Punjab Education Sector Project 135. Pakistan - Second Phase of the Punjab Barrages Improvement Project 136. Pakistan - Social Safety Net Technical Assistance Project 137. Panama - Health Equity and Performance Improvement Project 138. Panama - Metro Water and Sanitation Improvement Project 139. Papua New Guinea - Second Mining Sector Institutional Strengthening Technical Assistance Project 140. Paraguay - Water and Sanitation Sector Modernization Project 141. Philippines - Chiller Energy Efficiency Project 142. Philippines - Participatory Irrigation Development Project 143. Philippines - Social Welfare and Development Reform Project 144. Poland - Post Accession Rural Support Project 145. Poland - Second Road Maintenance and Rehabilitation Project 146. Rwanda - Electricity Access Scale-up and SWAp Development Project 147. Rwanda - Land Husbandry, Water Harvesting, and Hillside Irrigation Project 148. Rwanda - Sustainable Energy Development Project (GEF) 149. Senegal - Participatory Local Development Project 150. Senegal - Quality Education for All Project 151. Senegal - Second Sustainable and Participatory Energy Management (PROGEDE II) 152. Serbia - Delivery of Improved Local Services Project 153. Sierra Leone - Integrated Public Financial Management Reform Project 154. Sierra Leone - Institutional Reform and Capacity Building 155. Sierre Leone - Mining Technical Assistance Project 156. Solomon Islands - Health Sector Support Program Technical Assistance Project 157. Solomon Islands - Sustainable Energy Project 29 158. Sri Lanka - Higher Education for the Twenty First Century Project 159. Sri Lanka - Second Community Development and Livelihood Improvement Project 160. Tajikistan - Public Financial Management Modernization Project 161. Tajikistan - Community & Basic Health Project 162. Tanzania - Accountability, Transparency, and Integrity Project 163. Tanzania - Agricultural Sector Development Project 164. Tanzania - Science and Technology Higher Education Project 165. Tanzania - Strategic Cities Project 166. Tanzania - Sustainable Management of Mineral Resources 167. Tanzania - Transport Sector Support Project 168. The Gambia - Integrated Financial Management and Information System Project 169. Timor Leste - Health Sector Strategic Plan Support Project 170. Timor Leste - Planning and Financial Management Capacity Building Program Project 171. Togo - Financial Sector and Governance Project 172. Uganda - Health Systems Strengthening Project 173. Uganda - Local Government Management and Services Delivery (Adaptable Program Loan) Project 174. Uganda - Transport Sector Development Project 175. Uganda - Kampala Institutional and Infrastructure Development Project 176. Ukraine - Equal Access to Quality Education in Ukraine Project 177. Ukraine - Public Finance Modernization Project 178. Ukraine - Power Transmission Project in Support of the Energy Sector Reform and Development Program 179. Uzbekistan - Bukhara and Samarkand Sewerage Project 180. Uzbekistan - Public Finance Management Reform Project 181. Uzbekistan - Second Phase of the Basic Education Project 182. Vietnam - Central North Region Health Support Project 183. Vietnam - Da Nang Priority Infrastructure Investment Project 184. Vietnam - GEF-Coastal Cities Project 185. Vietnam - PCB Management Project 186. Vietnam - Second Northern Mountains Poverty Reduction Project 187. West Africa - Regional Fisheries Program Project 188. West Bank and Gaza - Capacity-Building for Palestinian Economic and Regulatory Institutions Project 189. West Bank and Gaza - Municipal Development Program Project 190. Yemen - Fourth Phase of the Social Fund for Development Project 191. Yemen - Integrated Urban Development Project 192. Yemen - Secondary Education Development and Girls Access Project 193. Yemen, Republic of - Social Welfare Fund Institutional Support Project 194. Yemen, Republic of - Social Fund for Development IV 30 Annex C Institutional Capacities and Their Contributing Characteristics Three broad areas of institutional capacity and their contributing characteristics can help assess institutional capacity needs, inform design, and measure progress toward results for a capacity development strategy or intervention. Strength of Stakeholder Ownership How strong is stakeholder ownership of the development goal? Ownership comprises formal and informal political, economic and social forces that determine the priority that government, civil society, and the private sector give to a development goal. Institutional Areas Covered Characteristic Commitment Social and political leaders consistently and regularly support the develop- of social ment goal through public statements, decisions, votes, and other actions and political demonstrating their commitment and accountability for achievement of the leaders goal. Leaders demonstrate sound ethics and integrity related to the goal, faithfully honoring their pledges of support and not engaging in activities or missions that would undermine progress toward the goal. Compatibility The social norms and widespread beliefs that shape the behavior of local of social norms stakeholders are compatible with the development goal. and values Stakeholder Priority-setting processes related to the development goal include oppor- participation tunities for all stakeholders to express opinions freely and without fear of in setting repression, and these opportunities are communicated with adequate time priorities for stakeholder engagement and response. Stakeholders actively participate through these established mechanisms, and government and other public service entities are responsive to the expressed views of civil society and other relevant parties. Stakeholder Stakeholders know their rights related to the development goal, claim those demand for rights, and communicate their grievances and proposals for change to accountability the government and legislature. If public officials and other public service providers fail to meet expectations about the achievement of the goal or are using allocated resources for unintended purposes, stakeholders freely hold them accountable for their conduct and performance. Stakeholders organize and collectively appeal unfavorable decisions concerning the goal. Stakeholders demand and use instruments of accountability that are available to the public, such as government scorecard information. Transparency Government and other public service entities provide accurate, relevant, of information verifiable, and timely information about the development goal to all to stakeholders stakeholders. They also explain actions concerning the goal in terms that stakeholders can use to participate in setting priorities, monitoring progress, and evaluating actions of public officials responsible for the goal. Any information influencing decisions related to the goal is fully disclosed to stakeholders. 31 Efficiency of Policy Instruments How efficient are policy instruments in guiding stakeholder behavior toward achieving the development goal? The efficiency of policy instruments comprises the administrative rules, laws, regulations, standards, and other formal incentives that a society uses to guide stakeholder actions to achieve its development goals. Institutional Areas Covered Characteristic Clarity in The rights and responsibilities of stakeholders related to the development defining goal are clearly defined and communicated. Stakeholders share a common rights and understanding of the policy goal and the targets of any specified regulations. responsi- The authorities and processes concerning the policy instrument are clearly bilities identified and communicated to stakeholders. Stakeholders understand the objectives, rights and responsibilities prescribed by the policy instrument. Consistency Policy instrument related to the development goal is consistent with policy instruments for other goals. The objectives, rights and responsibilities established by the policy instrument related to the goal do not conflict with those for other goals. Legitimacy Policy instrument is formulated based on an informed, transparent, participatory, and deliberate process. The instrument conforms to recognized principles and standards and is perceived as desirable and appropriate within the local system of norms, values, beliefs, and definitions. Stakeholders and unaffiliated expert reviewers perceive the actions and sanctions prescribed by the policy as fair. Stakeholders’ rights to appeal are assured by the policy instrument. Incentives for Policy instrument facilitates desired economic and social exchange activities compliance related to the development goal, minimizes uncertainty, time, information and other costs to participants in these transactions, and provides sanctions for non-compliance. Possible measures include: share of targeted stakeholders who elect self-policing, self-auditing, or self-disclosure options provided by the policy instrument; share of targeted stakeholders who comply with provisions of the policy instrument. Ease of Policy instrument is administratively easy for the regulatory or governing administration body to implement. Implementation incurs only those administrative costs required to achieve the intended policy outcomes related to the goal. Administrative tasks are defined and allocated with consideration of duty bearers’ competencies, existing responsibilities, and available resources. Risk for Policy instrument minimizes unintended negative impacts. Targeted negative beneficiaries, non-beneficiaries, and other non-regulated stakeholders do externalities not experience foreseeable unintended costs or ill effects as a result of the policy instrument. Suitable Policy instrument is predictably flexible in addressing varying situations. The flexibility policy instrument allows for timely revision when the underlying social and political circumstances have changed. The policy instrument is explicit in defining the conditions under which its terms should be revised. Resistance to Policy instrument minimizes opportunities for corruption, rent seeking, and corruption regulatory capture; includes mechanisms to monitor and report corruption; and provides credible and enforceable penalties for corrupt behavior. The policy instrument does not reflect the efforts of vested interests to manipulate the economic and/or legal environment to secure undue privileges or compensation at the expense of the greater public good. 32 Effectiveness of Organizational Arrangements How effective are organizational arrangements in achieving the development goal? The effectiveness of organizational arrangements comprises the systems, rules of action, processes, personnel, and other resources that state or non-state stake- holders bring together to achieve a development goal. Institutional Areas Covered Characteristic Clarity of The vision and mission of the organization/group are strongly aligned with mission the goal and articulated, providing its members with points of reference for formulating strategy, making decisions, and gaining commitment from management, staff, and other stakeholders to work toward the goal. Relevant stakeholders recognize the mandate of the organization Achievement The organization/group consistently seeks and achieves outcomes that lead of outcomes directly to the goal. Operational The strategies, inputs, processes, and technology of the organization/group efficiency are managed to optimize the quantity and quality of output, relative to the cost of accomplishing its support for the goal. Financial The organization/group secures the funds .needed to cover its operating viability and costs sustainably and allocates funds according to business needs. Financial probity management includes the tracking of expenditures on a functional basis to inform future budget forecasting, decision-making, and accountability reporting. Organizational systems and practices support and enforce transparent financial management, including the reporting of externally verified accounts, to ensure that the organization’s resources are allocated effectively to achieve its stated goal. Communi- The organization/group seeks and obtains the support of stakeholders for its cations and work related to the development goal. Organizational decision-making and stakeholder operational processes involve consultations and proactive communications relations with appropriate stakeholders. Adaptability The organization/group regularly monitors its internal and external environment for information relevant to the development goal and is proactive in adapting its strategy accordingly. The organization encourages innovation in strategies, processes and products; manages knowledge; rewards performance; and creates and/or adapts to new technologies. 33 Annex D Overview of the Institutional Capacity Indicators Database The Institutional Capacity Indicators Database can help task teams to monitor, evaluate, and report tangible results for their capacity development programs. The database is a searchable catalogue of real-world capacity characteristics and their indicators. In this way, project teams can break institutional capacities down into observable and measurable units to inform program design, monitor and measure changes, and manage results. http://wbicdrf.worldbank.org Background Task teams often need concrete results for capacity development initiatives, to show the viability of their efforts and accountability to stakeholders and donors. However, evaluation of capacity development activities often focuses on measur- ing outputs rather than outcomes. Also, the absence of appropriate indicators and failure to effectively track them limit the possibility for task teams to make needed, timely adjustments to their programs. WBI developed the Institutional Capacity Indicators Database as a convenient tool for task teams to find examples of indicators and measures for various institutional capacity challenges their projects face. The database features examples of indica- tors from a review of development databases and approximately 200 existing and closed World Bank projects across sectors and regions. The database can make it easier for task teams to take a results-focused approach to capacity development programs and improve project design, implementation, and management. 34 Uses and Benefits Task teams can search the Institutional Capacity Indicators Database to: • Identify characteristics of institutional capacity to inform program design • Identify indicators of those characteristics for periodic data collection to monitor and evaluate institutional change • Prioritize characteristics according to the most needed results to better target interventions • Expand their understanding of results management for capacity development Real-World Example Doreen is a task team leader within an urban development sector who focuses on public sector governance. She is working on a capacity development program with the goal of improving public services in targeted urban areas. She initially planned to focus primarily on organizational capacity development. However, after working with stakeholders and using the Institutional Capacity Indicators Database to inform the needs assessment, she adopted a more comprehensive approach to institu- tional capacities that also targeted the capacity area of stakeholder ownership. Through the database, Doreen understood how constraints to stakeholder own- ership could thwart the program’s ability to achieve locally owned results. She searched the database for examples of capacity characteristic outcome indicators and their measures to help inform program design. Together with stakeholders, Doreen also prioritized the characteristics according to the most needed results. By working with the database, Doreen was able to better allocate limited funds for the capacity development program and to support needed results in a previously neglected capacity area. Program planning became a more transparent and collab- orative process with stakeholders in identifying capacity characteristics, indicators, and data measures for local M&E practice. An example of an institutional capacity characteristic, its indicator and measure that Doreen will use in the capacity development program is charted below. Sector Urban Development Development Goal Improve public services in targeted urban areas Institutional Capacity Stakeholder ownership Area Characteristic Stakeholder participation in setting priorities Indicator Participation of community stakeholders in decisions regarding local public services Measure Participation rate of poorest and vulnerable community members in planning meetings Data Source Management Information System of Oversight Consultant, World Bank 35 References Organisation for Economic Co-operation and Development (OECD). 2005. The Paris Declaration on Aid Effectiveness. Paris. Organisation for Economic Co-operation and Development (OECD). 2006. Survey on Monitoring the Paris Declaration: Overview of the Results. Paris. Otoo, S., Agapitova, N., and Behrens J. 2009. The Capacity Development Results Framework: A Strategic and Results-Oriented Approach to Learning for Capacity Development. World Bank Institute, Washington D.C. Taylor, P. and P. Clarke. 2008. Capacity for a Change. Institute of Development Studies. Sussex. World Bank Independent Evaluation Group (IEG). 2005. Capacity Building in Africa: An IEG Evaluation of World Bank Support. Washington, D.C. World Bank Independent Evaluation Group (IEG). 2006. Annual Review of Development Effectiveness. Washington, D.C. World Bank Independent Evaluation Group (IEG). 2008. Using Training to Build Capacity For Development: An IEG Evaluation of the World Bank’s Project-Based and WBI Training. Washington, D.C. 36 The World Bank Institute’s Capacity Development and Results practice prepared this report. Dawn Roberts and Cristina Marosan Ling led the report team. Dawn Roberts and Natalia Agapitova led the indicator review work. Samuel Otoo provided overall guidance. Sharon Fisher provided editorial support. Design: Sharon Fisher, World Bank Institute Photos from The World Bank collection. Curt Carnemark: Cover center; annex left; back cover center and right. Arne Hoel: Inside front cover left and right. Edwin Huffman: Annex right; back cover left. Simone D. McCourtie: Cover right; annex center; inside back cover left. Samuel Otoo: Cover left; inside back cover right. Ray Witlin: Inside front cover center; inside back cover center. 37 A review of appraisal documents for approximately 200 projects in the World Bank’s lending portfolio provides important insights for strengthening the results measurement of capacity development components. Missed opportunities found in the projects’ results frameworks demonstrate the need for a more systematic approach for monitoring and assessing results. At the same time, isolated good practices confirm the feasibility of using the World Bank Insitute’s Capacity Development and Results Framework to address this need. Findings from the review describe why and how this framework could help project teams more effectively measure and communicate results of capacity development interventions. 38