94976 Selectivity in Country Strategies: The Evidence Abbreviations and Acronyms AAA Analytical and Advisory Assistance CASCRR Country Assistance Strategy Completion Report Review CAE Country Assistance Evaluations CPE Country Program Evaluation CPF Country Partnership Framework EAP East Asia Pacific ECA Europe and Central Asia IBRD International Bank for Reconstruction and Development IEG Independent Evaluation Group IFC International Finance Corporation IDA International Development Association FCS Fragile Conflict Situation LAC Latin America and the Caribbean LIC Low-Income Countries MENA Middle East and North Africa MIGA Multilateral Investment Guarantee Agency OPCS Secretariat, Operational Policy and Country Services PPP Public-Private Partnership RAP Results and Performance of The World Bank Group SAR South Asia Region VPU Vice Presidency Unit WBG World Bank Group This paper was co-authored by Xiaolun Sun, Luis Alvaro Sanchez, Carla Paecz, Shoghik Hovhannisyan, Xue Li and Geeta Batra. Assistance was provided by Cecilia Kern and Corky De Asis. Director-General, Evaluation : Ms. Caroline Heider Director, Independent Evaluation Group, IEGCC : Mr. Nick York Manager, IEGCC : Ms. Geeta Batra i Contents 1. Introduction ........................................................................................................................... 1 2. How is Selectivity Defined in CASs? ................................................................................... 2 3. Trends in Selectivity ............................................................................................................. 4 4. Selectivity in Consecutive CASs .......................................................................................... 8 5. Does Selectivity Matter for Development Outcomes? ......................................................... 9 6. Findings from IEG’s Country Program Evaluations (CPEs) .............................................. 11 7. Lessons from CPEs for Improving Selectivity ................................................................... 12 8. Conclusions ......................................................................................................................... 14 Boxes Box 2. 1. Broad Categories of Selectivity in 54 CASs ............................................................. 3 Tables Table 5. 1. Regression results for the impact of selectivity on IEG’s Overall Outcome Ratings of CASCR Reviews ................................................................................................................ 10 Figures Figure 2. 1. Discussions of Selectivity within the CPS, FY 09-13 ........................................... 3 Figure 2. 2. CASs with Multiple Definitions of Selectivity ..................................................... 4 Figure 3. 1. Distribution of CAS Objectives 5 Figure 3. 2. Selectivity across Income Groups and Regions – by Number of Objectives ........ 6 Figure 3. 3. Distribution of Sector Boards in CASs ................................................................. 6 Figure 3. 4. Selectivity across Income Groups and Regions – by Number of Sectors ............. 7 Figure 3. 5. CAS Selectivity, FY01-14 ..................................................................................... 7 Figure 4. 1. Direction of Changes in the # of Objectives in Consecutive CASs, FY03-FY14……………………………………………………………………………………8 Appendixes Appendix A. CAS Selectivity Database ................................................................................. 15 Appendix 2: Consecutive CASs.............................................................................................. 20 Appendix 3: CAEs and Selectivity ......................................................................................... 22 1 1. Introduction 1.1 The World Bank Group (WBG) has adopted a new strategy which sets two ambitious goals of ending extreme poverty and promoting shared prosperity. To operationalize the twin goals, the WBG is developing a more evidence-based and selective country engagement model, the Country Partnership Framework (CPF). The Bank Group’s activities in any country will be at the intersection of what the Systematic Country Diagnostic reveals, the government’s own development priorities and the WBG’s comparative advantage (OPCS). While the CPF is sharpening the WBG’s focus on strategic selectivity in its country programs, the issue is longstanding. For example, OPCS’s 1998 Country Assistance Strategies: Retrospective and Outlook (CAS Retro I) devotes considerable attention to the issue of strategic selectivity, providing a quantitative analysis of sectoral selectivity, defining strategic selectivity as a matter of qualitative judgment based on three criteria, and recommending specific actions by OPCS, the Networks, and country teams in order to strengthen strategic selectivity and focus in the CASs. The two subsequent CAS Retrospectives (2000 and 2003) report an improvement in the treatment of selectivity in CASs. OPCS noted that although most CASs considered one or more selectivity dimensions outlined in CAS Retro I, many did not fully distill their implications for the design of WBG program and that the selectivity improvement was mostly in discussing the rationale for choosing one instrument over another. With the adoption of results based CASs, selectivity is viewed as “a natural by-product of the careful construction of a results framework” (CAS Retro IV, 2005), and the CASs are judged as having become “more selective and focused” and “broadly satisfactory in prioritizing areas for Bank support” (CAS Retro V, 2009). 1.2 Selectivity, or its lack thereof, has often been identified by IEG as a key factor affecting the outcomes of the WBG’s country engagement, although IEG has never provided an explicit definition of selectivity. In one of its earliest Country Assistance Evaluations (CAE) - the 2000 Tanzania CAE, IEG indicated that the lack of emphasis on strategic selectivity in donor assistance was a major weakness in the Bank’s support program. More than a decade later, IEG discussed at length in its latest country program evaluation - the 2014 Tunisia Country Partnership Evaluation (CPE) - why the unsatisfactory outcomes of the WBG program were attributable to a lack of selectivity. Similarly, IEG has frequently raised the same concern in its reviews of CAS Completion Reports (CASCR-Rs). It was first alluded to in the 2004 Armenia CASCR-R, the second CASCR-R prepared by IEG, and formally discussed in the 2005 Cambodia CASCR-R, which rated the outcome of the Bank program as unsatisfactory. In the last three years (FY12-14), issues with selectivity have been observed in 45 percent of the programs rated moderately unsatisfactory or worse and 34 percent of those rated moderately satisfactory or above. 1.3 Clearly, strategic selectivity has been a concern within the WBG; yet it poses a challenge both conceptually and operationally. The 2005 OPCS Guidelines to Staff for CAS Products provided guidance on stakeholder consultations in order “strengthen the strategic selectivity of the program on the basis of the Bank’s comparative advantage”. The 2012 OPCS Guidelines for CAS Products again highlighted the importance of strategic selectivity in the context of the aid effectiveness agenda, but provided no further 2 guidance on how to define selectivity or how to apply the concept in developing WBG country programs. The new CPF directive was recently discussed by the Executive Board and a new guidance remains to be implemented. 1.4 This paper is a first attempt at piecing together the various strands of evidence in order to understand the role and the practice of selectivity in the WBG’s country strategies, and explores the link between selectivity and country program outcomes. It reviews selectivity in 105 CASs, including Country Partnership Strategies, during FY09- 13. It also provides a synthesis analysis on selectivity issues of 22 CAEs, including Country Partnership Evaluations (CPEs), conducted by IEG during FY05-14. The findings demonstrate that selectivity matters for the overall development outcome of CASs while controlling for other variables such as country ownership, results framework, and GDP per capita. Moreover, the estimations indicate that selectivity is more important in countries with high levels of extreme poverty. Finally, the paper concludes with the key lessons and issues for further research. 2. How is Selectivity Defined in CASs? 2.1 Defining selectivity as “a matter of qualitative judgment”, OPCS’s CAS Retro I (1998) proposed three criteria for exercising selectivity in WBG country programs: (i) potential magnitude of impact; (ii) likelihood of country action; and (iii) additionality of Bank contribution. The relative cost of Bank involvement in an activity or sector was also highlighted as a fourth factor for guiding strategic selectivity in CAS programs. 2.2 Over the last five years, the WBG has paid increasing attention to selectivity. Most of the CASs (62 out of 105) discuss selectivity when specifically addressing the upcoming country strategy, and there is an upward trend in the frequency with which selectivity is considered. In many cases, selectivity is one of the principles of engagement (strategic principles) outlined in country strategies, along with partnership and flexibility; in other cases, selectivity is presented as the way in which the WBG chooses to intervene, but without further elaboration of what this means. 1 Overall, the vast majority of the CASs that discuss selectivity (54 out of 62) provide a definition, or a filter, of selectivity for choosing engagement areas in the upcoming WBG country strategies. 1All CASs are aligned with client country strategies and focus on priority areas as identified in the government strategies. These are excluded from the definition of selectivity here. 3 Figure 2. 1. Discussions of Selectivity within the CPS, FY 09-13 2.3 Definitions of strategic selectivity have been typically based on qualitative judgment about what is needed in a particular country context, which has led to a focus on different aspects of selectivity, as well as markedly divergent criteria for exercising selectivity across the WBG. For instance, in the 54 CASs in which selectivity is considered, no fewer than 30 different definitions are formulated. Often, a CAS defines selectivity in more than one dimension. The most common dimensions are donor involvement, sector involvement, WBG comparative advantage, and WBG past experience; each of which is mentioned by more than a third of the CASs. The least common criteria include potential for scaling up, leveraging institutional change, and corruption risks, among others. Box 2. 1. Broad Categories of Selectivity in 54 CASs • Measurement of selectivity based on the number or nature of the objectives, themes, sectors, and financing instruments, or the number and size of projects (20 percent of CASs). • Donor activities to ensure proper division of labor or pooling of resources (17 percent of CASs). • Client needs, priorities, commitments, and capacity, as well as corruption risks (20 percent of CASs). • WBG capacity to deliver based on past experience; the WBG’s comparative advantage, or policy and financing constraints (30 percent of CASs). • Approaches for impact, including a diverse set of considerations such as institutional change, capacity development, potential for scaling-up, sustainability without aid, mutually reinforcing multi-sectoral projects, regional integration, time horizon for results, and addressing fragility drivers (14 percent of CASs). 4 2.4 In general, the CASs emphasize greater selectivity when the WBG is a small player (e.g., Pacific islands), where the development needs are great compared to resources available (e.g., Niger), when the Bank program is too large to handle and reducing the number of projects has become an urgent priority (e.g., Pakistan), and in large middle income countries where a more sophisticated use of WBG support is demanded (e.g., India). Depending on the WBG’s experience in a particular country, these filters of selectivity may be applied in different ways. In some cases, the WBG chooses to engage where other donors are absent to avoid duplication of efforts; in other cases, the WBG only engages where other donors are already investing to ensure a meaningful impact. 2.5 Selectivity is often applied across multiple categories presented in Box1. In only 8 out of the 54 CASs is there a single filter for exercising selectivity, while in 3 cases all five categories of selectivity criteria – thus a wide range of different considerations – are used to rationalize what the WBG would do when engaging with clients. Two of the selectivity filters – client-related concerns and transformational approaches – are never utilized alone, but always used in conjunction with other filters. Figure 2. 2. CASs with Multiple Definitions of Selectivity 2.6 Figure 2 shows the degree of overlap in each category of selectivity filter. For example, the WBG’s capacity to deliver results is the only selectivity factor in five CASs, but is considered in conjunction with another filter in 10 CASs, with two other filters in 13 CASs, with three other filters in 12 CASs, and as one of all five types of filters in three CASs. In 31 percent of the CASs, three criteria are applied and in 26 percent of the cases, 4 criteria are used. 3. Trends in Selectivity 3.1 Earlier discussions of strategic selectivity tended to have a strong sectoral orientation. CAS Retro I (1998) and II (2000), for example, assessed CAS selectivity on the basis of the sectoral coverage and concentration of each CAS programs against the fourteen Network Family areas. Over time, it was recognized that ‘activity counting is a rough instrument for analyzing selectivity”, not least because many WBG activities are 5 multi-sectoral in nature. CAS Retro V (2009), in particular, argued that “rather than the number of sectors or themes where the Bank plans activities … the number of CAS results, and in particular, CAS outcomes, is a good first indicator of selectivity.” In addition, selectivity has also been viewed as an intra-sectoral issue – it may be more cost effective to have a few big projects than a multiplication of small projects to support the objectives in a particular sector. 3.2 In this study, these two basic indicators of selectivity are examined. Clearly, both indicators leave much to be desired, as is often the case when one attempts to quantify complex concepts; nevertheless, they remain conceptually relevant and operationally applicable moving forward. These two indicators are the number of CAS objectives or outcomes in a country strategy and the number of sectors in which the WBG engages during a CAS period. 3.3 CAS Objectives. During FY05-14, the WBG prepared 201 CASs for 107 countries. 2 The number of CAS objectives, as summarized in the annex CAS results matrix (but often referred to as outcomes), are counted for each CAS as an indication of the program’s scope. Using this measure, a CAS program that pursues a large number of different objectives would be considered less selective than one that focuses on a more limited set of issues. As shown in Figure 3, the number of objectives varies widely from CAS to CAS, ranging from zero (Mexico 2008) to 53 (Morocco 2010). The average number of objectives is 15, while 50 percent of the CASs had between 10 and 18 objectives. Figure 3. 1. Distribution of CAS Objectives 3.4 Measured by the number of objectives, there is little variation in CAS selectivity across countries in different income groups, except in FCS where the scope of the WBG programs tends to be more limited than in other countries – the average number of CAS objectives is 13, compared to 15-16 elsewhere. However, within the group of FCS countries, considerable variations exist as reflected in the larger size of the 50-percent box in Figure 4 below. 3 Across regions, there are more variations in CAS selectivity: the WBG appears to pursue a more limited set of objectives in LCR and SAR (12 objectives 245 countries had one CAS, 31 countries had two CASs, and another 31 countries had three CASs. 3The box indicates the relative concentration of 50 percent of the observations. The bigger the box, the more dispersion there is among the observations. 6 per CAS) than in MENA and ECA (15 objectives). There are some notable outliers with substantially higher number of objectives in many regions and income groups. Figure 3. 2. Selectivity across Income Groups and Regions – by Number of Objectives 3.5 Sectors Represented in CASs. All WBG activities, including both lending and knowledge services, are mapped to a single sector board. All sector boards that had at least one activity mapped to it during a CAS period are counted for each completed CAS. Ongoing CASs are excluded because a complete count of CAS activities cannot be performed. On average, the Bank provided financial and/or knowledge support to client countries in 12 different sectors during the strategy period, although in 16 cases the Bank was involved in all 15 sectors, while the 1999 Gabon CAS chooses to engage in only two issue areas. The Bank’s sectoral engagement is clearly skewed toward the high end of the scale: in more than 50 percent of the CASs, Bank activities span between 11 and 14 different sectors. Figure 3. 3. Distribution of Sector Boards in CASs 3.6 As measured by the number of sector board coverage in each CAS, the Bank’s program appears to be less selective in FCS and LIC countries, where the median number of sector boards is 13, than in other countries where the comparable number is 11. This 7 outcome may possibly reflect the fact that in poorer countries, where the development needs are greatest and wide-ranging, the Bank’s assistance is requested more extensively and broadly, and the Bank plays the key integrator’s role to bring together all other stakeholders. Across regions, some variations can also be observed: selectivity is higher in LCR (median number of sector boards is 10) than in EAP and SAR, where it is 14. Figure 3. 4. Selectivity across Income Groups and Regions – by Number of Sectors 3.7 Using these two crude measures of CAS selectivity, Figure 7 traces the evolution of the WBG’s practice of selectivity in country strategies between FY01 and FY14. Measured by the number of CAS objectives, there appears to be two distinct periods in selectivity trends. The first period (FY05-09) corresponds to that examined under CAS Retro V (2009), which observed a dramatic decline in the scope of WBG country programs that culminated in an unusually low number of objectives for the 20 CASs approved in FY09. Thereafter, the CAS scope expanded for a couple years, coinciding with the crisis, before declining again. There is no discernible change in the sector spreads of WBG country programs. Figure 3. 5. CAS Selectivity, FY01-14 3.8 Clearly these static measures of cross country selectivity do not reflect the evolution of selectivity in a given country program over time, and also do not capture the heterogeneity across countries and WBG programs. 8 4. Selectivity in Consecutive CASs 4.1 Between FY03 and FY14, there are 155 consecutive CASs for 62 countries. In 23 countries, the WBG’s country strategy has become more selective over time. Three countries (Niger, Vietnam and Zambia) with three successive CASs, in particular, have seen a continuous reduction in the number of CAS strategic objectives from one CAS to the next. In 18 countries, however, the trend is the opposite, with the scope of WBG support programs expanding in the more recent CASs. The three succeeding CASs for Benin and Indonesia, for example, all consistently increased the number of objectives in the newer strategies. In addition, most of the countries that have three consecutive CASs (21 out of 31) saw a reversal in the scope of the WBG’s country strategies: in 12 countries the number of objectives expanded in the latest CAS following a contraction in a previous CAS; in 9 countries there was an effort to rein in the expanding scope in the older CASs. Figure 4. 1. Direction of Changes in the # of Objectives in Consecutive CASs, FY03-FY14 4.2 Among the 48 instances of decreasing the number of CAS objectives, many reflect genuine efforts by the country teams to scale down the scope of the Bank program: the number of lending and non-lending operations, as well as the sector coverage of Bank engagement, all declined in tandem. 4 However, there are also a number of cases where improving CAS selectivity was accompanied by an expansion of the Bank program. 5 A closer examination of the consecutive CASs reveals some important findings. 4.3 First, there is a close connection between the analysis of selectivity in the completion reports (CASCRs) and how selectivity is treated in the follow-up country strategy. The vast majority of these CASs adopt selectivity as a guiding principle going forward and nearly all of them have a CASCR of a previous strategy that highlight selectivity and/or related issues (e.g., leverage and strategic focus) as a key factor for program success. To a great extent, such a connection arises because of the continuity across strategies, which allows for increasing clarity over time of what is truly important and how to prioritize. 4 Examples include Burkina Faso, Cape Verde, Bosnia-Herzegovina, Croatia, Guatemala, Macedonia, Moldova, Niger, and Nigeria. 5 Examples include Armenia, Brazil, Malawi, Mexico, Vietnam and Zambia. 9 4.4 Second, despite dedicated efforts to improve selectivity, some teams have found that transforming an indiscriminate program into a selective one is not easy and may take longer than expected to implement. This is especially true where the circumstances encourage expanding engagement (e.g., post conflict) and where development needs are overwhelming. What is clear is that efforts to improve selectivity through “relabeling” or “regrouping” of the CAS objectives without a corresponding streamlining of the supporting programs, both lending operations and knowledge services, is not sustainable. 4.5 Third, selectivity is not only important at the strategy level, but also essential at project level, especially for adjustment lending operations. Excessive optimism about the client’s commitment and/or capacity is often linked to overly complex project design with an unwieldy large scope and unrealistic implementation schedule. Sometimes, the drive to achieve strategic selectivity by doing fewer, larger operations that leverage Bank and partner resources has led to a loss of operational selectivity due to expanded scope of individual projects. 6 5. Does Selectivity Matter for Development Outcomes? 5.1 Amongst the 41 Completion Reports (CASCRs) prepared in FY13 and FY14 and their corresponding follow up strategies, 24 of them (roughly 60 percent) refer to selectivity as a driver of results. 7 For example, completion reports for Georgia and Rwanda, explicitly asserted that selectivity mattered for the satisfactory delivery. 8 5.2 Rwanda’s Completion Report states: “Having a highly selective program helped to focus the design of the CAS in key strategic areas where the WBG would add value. The CAS program targeted few areas, essential to Rwanda’s continued growth and development aspirations. Government’s strong commitment to effective use of aid through increased donor coordination and country ownership, further helped achieve selectivity.” IEG’s review of the Completion Report concurred with the relevance of selectivity for the delivery of results and added that it helped selectivity that the country has clear ideas as to where the WBG should operate. 5.3 Georgia’s Completion Report explains, “The selectivity and complementarity between IBRD, IFC and MIGA has been effective. WBG funding was channeled to maximize comparative advantage, technical knowledge, and complementarities. The selectivity in the Bank’s program allowed for an approach that balanced concerted focus on strengthening the effectiveness of public expenditure and administration through AAA and DPO policy reforms, together with targeting specific investments in infrastructure and regional development programs. This complemented IFC support, which continued to focus on strengthening the financial system and supporting enterprises in agribusiness, as well as MIGA’s engagement in the financial sector.” “Program selectivity has been a strong feature 6 This observation applies to both DPLs and SILs and does not intend to compare the two instruments. By IEG’s ICR-R ratings, DPLs generally outperform SILs, although the methodology for assessing the two types of projects is different, thus not comparable. 7 This number most likely underestimates the role selectivity plays, as only cases where selectivity is mentioned explicated are included. 8 China is the third satisfactory case. The Progress Report previous to the CR informed that the Government intended to apply the WBG CPS selectively with a focus on a core set of objectives. Thus, if not by design, de facto the strategy appears to have been selective. 10 of the program, facilitated by donor coordination.” IEG review concurred stated, “The one topic that resonates the most is selectivity. Georgia has a large volume of donor assistance and the Bank’s focus on a few key areas seems to have made a major contribution.” 5.4 Although it seems intuitively logical and anecdotal evidence from above suggests that a more selective and focused CAS would lead to better outcomes in a WBG’s country program, the expected positive relationship between CAS selectivity and outcomes has so far not been demonstrated empirically. The inherent difficulties in defining and measuring selectivity, as shown in previous sections, partially explain this gap. It should be noted in advance that the WBG’s program results are likely influenced by a wide range of factors, of which strategic selectivity is just one potential element. 5.5 Earlier IEG reviews had already identified country ownership, results frameworks of country programs as factors that influence country outcomes. (The Matrix System at Work, 2012; Results and Performance Report, 2013). We expand the analysis to include measures of selectivity based on the objectives/sectors, and average activity size, and controls for country income. Table 1 below reports the regression results. The overall outcome rating (dependent variable) is coded 1 for Highly Unsatisfactory, 2 for Unsatisfactory, 3 for Moderately Unsatisfactory, 4 for Moderately Satisfactory, 5 for Satisfactory, and 6 for Highly Satisfactory ratings. The second column “Model” reports the coefficients of the basic regression, which includes selectivity, activity size, ownership, results framework and GDP per capita as explanatory variables. CAS selectivity is captured by a dummy variable, which is coded 1 if the number of objectives or sector boards involved is smaller than their respective average values in all CASCR Reviews. We include another measure of selectivity, namely activity size, calculated as total commitment amounts divided by the total number of lending projects and non-lending Analytical and Advisory Activities (AAAs). Ownership is a categorical variable with a value of 1 for proper consultations and broad commitment by stakeholders in the design of strategy and sustained commitment during the program implementation and 0 otherwise. The results framework is a dummy variable with a value of 1 for an adequate results framework. The third column - Model with Interaction, augments the basic regression with an interaction term with selectivity and poverty and for the latter it uses a poverty headcount ratio at $1.25 a day (PPP) (% of population). These poverty data are consistent with WBG’s twin goals and are averaged for 2000-2012. Table 5. 1. Regression results for the impact of selectivity on IEG’s Overall Outcome Ratings of CASCR Reviews Independent Variables Model Model with Interaction Selectivity 1.96*** (.281) 1.164*** (.412) Activity size .017*** (.005) .015*** (.005) Ownership .987*** (.333) .973*** (.321) Results Framework .86** (.402) .993** (.392) GDP per capita .076*** (.026) .119*** (.03) Interaction term: selectivity and poverty .026*** (.01) Number of Observations 87 83 The numbers in parentheses are standard errors of the coefficients and (*) indicates significance level at 10, (**) at 5, and (***) at 1 percent. 11 5.6 The estimation results indicate that selectivity as measured by both—either objectives or sectors, and the activity size, is positively correlated with the overall outcome of CASs, while controlling for other factors. In general selectivity exercised through a set of important objectives, sound sector selection and large focused interventions are associated with positive country outcomes. All other variables included in the regression are positive and significant with ownership and results framework having higher magnitudes than others. These results remain robust with an inclusion of the interaction term for selectivity and extreme poverty headcounts. This result indicates that selectivity is more important in countries with high levels of extreme poverty. 6. Findings from IEG’s Country Program Evaluations (CPEs) 6.1 Out of the 22 CPEs and two multi-country evaluations of WBG engagement in low-income FCS countries and at state level in large countries that IEG conducted during FY05-14, 9 identify selectivity as a factor in the delivery of results. However, the analysis of how selectivity influences performance is sparse and relatively recent. Selectivity is also discussed sparingly in country programs that were implemented under difficult circumstances, including internal conflict or a high level of political instability, 9 and that have achieved satisfactory outcomes. 10 In the first group, the CPEs tend to deal mostly with the immediate challenges of conflict and/or instability; while in the second group selectivity does not emerge as a binding constraint to the delivery of results. Indeed, when the CPEs do address selectivity, it is mostly in the context of its absence, and thus as a cause for poor program outcome. 6.2 To address how selectivity affects CAS program success, IEG’s CPEs identify four main channels through which a lack of selectivity impacts WBG performance. • It increases the likelihood of extending Bank engagement into areas where the Bank has inadequate understanding of the risks or limited influence over the expected outcomes. The recent evaluations in FCS point out that most FCS strategies have not been underpinned by systematic analysis of the drivers of fragility, conflict, and violence, and consequently look no different from those in non-FCS countries. The Tunisia CPE highlights that in the face of generic risks of low government buy-in and a difficult political economy, the participation of the Bank was marginal in several engagement areas where the achievement of the stated objectives depended mainly on other actors. • It distracts a program from focusing on priorities even when these are identified. In Nepal, although the Bank strategy acknowledged two central issues (state- building and inclusion) for moving forward, these priorities were not used to exercise selectivity and the program remained focused on long-term, pre-crisis agenda, with the first project presented to the Board aiming at tertiary education. In contrast, the Afghanistan and Liberia country programs demonstrate that when 9 For example, Angola, Nepal, Madagascar, and West Bank and Gaza. 10 For example, Brazil, Cambodia, Egypt, Georgia, Indonesia, Mozambique, and Peru. 12 selectivity is driven by priorities clearly articulated in the government’s strategy and owned by the country, it leads to good outcomes. • It often leads to a dispersion of interventions. In the context of limited or fixed resources, such as in Sierra Leone, Cameroon and DRC, the Bank does many small and/or one-off projects which generally have a lower chance of success and are not able to make a significant impact. Diffused Bank efforts, broad coverage of issues, and a multiplication of activities are also associated with higher administrative cost. Moreover, once the Bank becomes involved in certain areas, it frequently finds it difficult to disengage, thus limiting the ability of subsequent strategies to exercise selectivity. High responsiveness to political pressures and in entirely demand-driven programs contributes to this dispersion. • It has a direct negative impact on project design. The widening scope of a strategy is not just reflected in a growing portfolio and pipeline, but also in the expanding scope of individual lending operations, especially policy lending instruments. Several CPEs identify the lack of selectivity in the Bank’s lending operations as a key factor for poor results delivery. 7. Lessons from CPEs for Improving Selectivity 7.1 IEG’s CPEs often recommend sharpening selectivity and focus in future CASs as a way to improve program outcomes, even those that do not articulate a clear role of selectivity in the delivery of results. It is also a recurrent theme in Board discussions, which often call for limiting the scope of WBG engagement. Five findings emerge from IEG’s evaluations. 7.2 A careful consideration of client country’s capacity. This is a recurrent concern, especially in FCS and poorer countries. The Angola CPE, covering a 15-year period that included a protracted internal war, emphasizes that “the impact of the assistance was dampened by limited capacity [of the government] to absorb Bank services and expectations that exceeded what was feasible to achieve.” The FCS evaluation stresses the same point. The concern over client capacity goes beyond the availability of skilled- staff and comprises the ability of the government to lead reform, where constraints range from a lack of knowledge, given the novelties of the approaches proposed by the international community, to a difficult political economy. As the Nigeria CPE notes, “as far as the Bank’s contribution is concerned, where there was political will to undertake reforms, the detailed design tended to follow the blueprint the Bank had laid out.” 7.3 Grounding interventions in solid analytical work. The Nigeria CPE points out that a key problem with a rapid program expansion lies in the risks engaging in areas where knowledge is thin. For the same reason, the Uganda CPE emphasizes supporting the government in the developing analytical frameworks to guide decisions. The Brazil and Bangladesh CPEs both call for long-term positioning of the analytical work and for looking past the vagaries of short-term demand by the client. In addition, given the importance of analytical work as a first step in decisions about country engagement, the Afghanistan CPE highlights the need to be strategic in the selection of AAA work and to ensure that it is aligned with critical priorities. 13 7.4 Taking a long-term view and sequencing interventions. The Timor-Leste CPE highlights the need for the WBG to “resist the temptation of attempting to do too much too fast” and “to be realistic about the situation on the ground and the capacities (and time-frames) for change and development.” The need to strike the right balance between interventions for long-term institutional development on the one hand and concrete measures to address short-term needs on the other is frequently emphasized. The FCS evaluation, for example, points out that while quick reconstruction operations are justified in the immediate post-conflict period utilizing emergency or nontraditional delivery mechanisms, continued reliance on expedient methods to implement projects would incur a heavy cost by delaying the establishment of a functioning state, which is required for longer-term development. On the difficult task of capacity building of relevant institutions, the Uganda CAE underscores the need for proper sequencing of reforms, sufficient time, and adequate risk analysis. Referring to the sudden introduction of government policy measures that may be desirable, but are out of line with the agreed medium and long-term strategic planning framework, it concludes that untimely sequencing of policy measures in an environment undergoing major reforms can lead to underperformance. 7.5 Understanding and properly factoring in the risks to the delivery of the expected results. Based on the observation that unanticipated consequences often arise and lead to weak outcomes when program design and choice of policy options do not adequately factor in risk issues in design and implementation, the Uganda CPE recommends reviewing the design and implementation options in the Bank’s assistance program with attention to their appropriateness to the country and to the risks associated with each option. The FCS evaluation goes a step further by arguing that the identification of risk factors must be accompanied by remedies. In particular, it points out that institution building requires legislation, adequate staffing, and leadership, which usually take longer to assemble than is anticipated in the original plan. A better understanding of the political context and a focus on basic risk analysis would help define a more realistic time horizon. 7.6 Taking into account the Bank’s comparative advantage. Often the Bank’s comparative advantage is discussed in the context of working with other development partners. However, the CAEs do not present clear criteria to assess comparative advantages, and the views expressed diverge. The Georgia CPE advised the Bank to stay within its established field of expertise such as it can realistically mobilize the skills needed to respond to client demand. The Bangladesh CPE sees the Bank’s comparative advantage in the preparation of analytical work and foresees a broader engagement in AAA as well as in complementary operations. Some evaluations, on the other hand, argue that comparative advantage can be built, to some extent, through AAA and technical assistance. 14 8. Conclusions 8.1 This paper reviews the WBG’s practices in strategic selectivity. It highlights the progress in thinking and in implementation, but also reveals important knowledge gaps that require further investigation. The overall findings that emerge from the data and analysis are: • The WBG is paying significant and increasing attention to selectivity. Many CASs explicitly address selectivity, articulate specific criteria for exercising selectivity, and draw lessons from the past for improving selectivity. • Selectivity (as measured by commonly used indicators such as number of objectives and sectors) has improved over time when considering consecutive CASs. In 23 out of 62 countries, the WBG’s country strategy has become more selective over time. • Selectivity is positively correlated with country program outcomes based on empirical analysis from CASCR Reviews. In addition to country ownership, good results frameworks, selectivity is correlated with better country outcomes. Moreover, selectivity is more important in countries with higher levels of extreme poverty, which emphasizes the importance of selectivity in achieving the twin goals. • There are no simple criteria to determine the size and composition of a program should be; that is, the number of objectives and/or the number of activities cannot be predetermined. What matters is the fit between design and context and henceforth selectivity is likely to be contextual. • While selectivity is multidimensional and complex, evidence from CPEs and the CASs, suggests that selectivity can be enhanced by paying careful attention to client country capacities, grounding all operations in solid analytical work, taking a long-term view and sequencing interventions, properly factoring in design and implementation risks to results delivery, and taking advantage of the Bank’s comparative advantage. The new Country Partnership Framework is consistent with the findings presented in this paper. 15 Appendix A. CAS Selectivity Database 1. Existing World Bank systems record a wealth of information that is mostly presented at the project level. In some cases, the information is not recorded at all, such as the overall and specific objectives that the WBG pursues during a CAS period. For the purpose of this study, two datasets were created - one of CASCR Reviews and the other of CAS Objectives – and linked up to provide a rich database for the present and future studies. The study also used the CASCR Review database of IEG’s evaluation on “Results and Performance of the World Bank Group 2013” (RAP2013) and poverty data from the World Development Indicators. The unit of observation is CAS. Scope of the database 2. The CASCR Review database contains 175 CASCR Reviews produced by IEG during FY03-14, which assessed the performance and achievements of CASs approved from FY99 to FY10. Although CASCR Review database provides information on performance - outcome ratings for each CAS program, they don’t present a complete picture of all the country programs implemented during these years because not all of the earlier CAS (prior to FY02) were reviewed, while the CASs approved in more recent years (after FY08) are still on-going. The CAS Objectives database includes 201 CAS approved during FY05-14. The objectives as they are stated in the annex results matrix or in the text are identified for each CAS. The CAS Objectives database does not include ISNs 11, Regional Strategies, and Progress Reports. 3. The RAP2013 CASCR Review database is based on IEG’s CAS Completion Report (CASCR) Reviews and Country Program Evaluations (CPEs) for FY08-FY13, covering countries in all World Bank Group (WBG) regions. Country strategies reviewed in this database were completed between FY07 and FY12. Out of 106 CASCRs reviewed by IEG during FY08-FY13, 56 corresponded to IBRD countries and 40 to IDA, excluding 10 FCS that have been taken as a separate category. 4. The first two databases were merged, resulting in a final dataset containing 98 CASs for which information is available on CAS objectives, portfolio, CAS outcome ratings, and client country capacities. There are 77 CASs that were prepared before the results-based CASs were introduced in FY05; without an official results matrix, their objectives cannot be accurately identified. There are also 103 ongoing CASs that will go through the CASCR-R process in the future. To conduct the regression analysis on the impact of the selectivity on CAS outcomes this data is further merged with the RAP2013 dataset, which contains variables on ownership, results framework and GDP per capita. Finally, to assess selectivity impact in poor countries this dataset is further merged with the poverty database of the World Development Indicators of the WBG. Table 2 shows the number of observations for all datasets used in this study and the sample for the regression analysis. 11 There are only a few exceptions when ISN Completion Reports have been reviewed with CASCRs. 16 Table 1. Number of CASs and CASCR-Rs by Different Databases Database Number of CASCR-RRs CASCR Review Database 175 CAS Objectives Database 201 RAP2013 CASCR Review Database 106 Poverty Database 125 Overlapped Database for Regression 87 Overlapped Database for Regression with selectivity and poverty interaction term 83 Variables used in the Regression Analysis 5. The merged database includes the following variables: • CAS Outcome Rating: 1- Highly Unsatisfactory, 2-Unsatisfactory, 2-Moderately Unsatisfactory, 4-Moderately Satisfactory, 5-Satisfactory, and 6-Highly Satisfactory. • CAS Selectivity: a dummy variable, which is coded 1 if the number of objectives or sector boards involved is smaller than their respective average values for all CASCR Reviews. • Activity size: Average size per activity, which is calculated as the total commitment amounts divided by the total number of lending projects and non-lending AAAs. • Ownership: a dummy variable which is coded 1 if there were proper consultations and broad commitment by stakeholders in the design of strategy and the commitment was sustained during the program implementation. • Results Framework: a dummy variable which is coded 1 if there was an adequate results framework. • GDP per capita in Purchasing Power Parity in constant 2011 International Dollars. • Interaction term with selectivity and poverty where poverty is defined as poverty headcount ratio at $1.25 a day (PPP) (% of population). Poverty data are averages for 2000-2012. 17 Table 2. List of Countries Included in the Regression CASCR Review Country Fiscal Year of CASCR Review Period Albania 2011 FY06-09 Armenia 2009 FY05-08 Azerbaijan 2011 FY07-10 Bangladesh 2010 FY06-09 Belarus 2008 FY02-06 Belarus 2013 FY08-11 Benin 2009 FY04-06 Benin 2013 FY09-12 Bhutan 2011 FY06-09 Bosnia-Herzegovina 2008 FY05-07 Bosnia-Herzegovina 2012 FY08-11 Brazil 2012 FY08-11 Bulgaria 2011 FY07-09 Burkina Faso 2010 FY06-09 Burundi 2013 FY09-12 Cameroon 2010 FY06-08 Cape Verde 2009 FY05-08 Chile 2011 FY07-10 China 2013 FY07-12 Colombia 2012 FY08-11 Congo, Democrat 2013 FY08-11 Congo, Republic 2013 FY10-12 Costa Rica 2009 FY04-08 Costa Rica 2012 FY08-11 Croatia 2009 FY05-08 Croatia 2013 FY05-08 Djibouti 2009 FY06-08 Dominican Republic 2010 FY06-09 El Salvador 2010 FY05-09 Ethiopia 2013 FY08-12 Gabon 2012 FY05-09 Gambia 2008 FY03-07 Gambia 2013 FY08-11 Georgia 2010 FY06-09 Guatemala 2009 FY05-08 Guatemala 2013 FY09-12 Guyana 2009 FY03-08 Honduras 2012 FY07-10 18 India 2009 FY05-08 India 2013 FY09-12 Indonesia 2013 FY09-12 Jordan 2012 FY06-11 Kazakhstan 2012 FY05-11 Lao PDR 2012 FY05-11 Lebanon 2011 FY06-09 Lesotho 2010 FY06-09 Macedonia 2011 FY07-10 Malawi 2013 FY07-11 Maldives 2008 FY00-07 Mauritania 2008 FY03-07 Mexico 2008 FY05-08 Moldova 2009 FY05-08 Montenegro 2011 FY07-10 Morocco 2010 FY06-08 Mozambique 2012 FY08-11 Nicaragua 2008 FY03-07 Nicaragua 2013 FY08-12 Niger 2013 FY08-11 Nigeria 2010 FY05-09 Pakistan 2010 FY06-09 Panama 2011 FY08-10 Paraguay 2009 FY04-08 Peru 2012 FY07 -11 Philippines 2009 FY06-09 Poland 2009 FY05-08 Romania 2010 FY06-Dec., 09 Russian Federation 2012 FY07-11 Senegal 2013 FY07-11 Serbia 2008 FY05-07 Serbia 2012 FY08-11 Sierra Leone 2010 FY06-09 South Africa 2008 FY00-06 Sri Lanka 2012 FY09-12 Tajikistan 2010 FY06-09 Tanzania 2011 FY07-10 Thailand 2011 FY03-09 Timor Leste 2013 CAS: FY06-08; ISN: FY10-11 Turkey 2012 FY08-11 Uganda 2010 FY06-09 19 Ukraine 2008 FY04-07 Ukraine 2012 FY08-11 Uruguay 2011 FY05-10 Uzbekistan 2012 FY08-10 Vietnam 2012 FY07-10 Yemen 2009 FY06-09 Zambia 2008 FY04-07 Zambia 2013 FY08-11 20 Appendix 2: Consecutive CASs 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 Decrease in Number of Objectives Albania x x Bulgaria x x Burkina Faso x x Burundi x x Cape Verde x x Congo, Democrat x x Croatia x x x Gabon x x Honduras x x Lao PDR x x Mauritania x x Niger x x x Nigeria x x Russian Federation x x Sierra Leone x x South Africa x x Tanzania x x Uganda x x Uzbekistan x x Vietnam x x x Zambia x x x Increase in Number of Objectives Bangladesh x x Belarus x x Benin x x x Bhutan x x Chile x x Dominican Republic x x Indonesia x x x Jordan x x x Kazakhstan x x Lebanon x x Lesotho x x Morocco x x Pakistan x x Panama x x Philippines x x Ukraine x x x Uruguay x x Decrease then Increase in Number of Objectives Armenia x x x Bosnia-Herzegovina x x x Brazil x x x Colombia x x x Guatemala x x x India x x x Macedonia x x x Malawi x x x Mexico x x x Moldova x x x Peru x x x Poland x x x Increase then Decrease in Number of Objectives Azerbaijan x x x 21 China x x x Ethiopia x x x Kyrgyz Republic x x x Montenegro x x x Mozambique x x x Serbia x x x Tajikistan x x x Ethiopia x x x Yemen x x x Little Change in Number of Objectives Nicaragua x x x Papua New Guinea x x Turkey x x x 22 Appendix 3: CAEs and Selectivity Outcome Selectivity matters How selectivity Recommend more Country Date Rating for results? matters explained? selectivity? Tunisia 2014 U Yes Yes Yes Brazil 2013 MS No No Yes World Bank Group Assistance to 2013 - Yes Yes Yes Low-Income Fragile and Conflict- Affected States Afghanistan 2013 MS Yes Yes Yes Liberia 2012 MS Yes No Yes Mozambique 2011 MS No No No Timor Leste 2011 MU Yes No Yes Cambodia 2010 MS No No No Nepal 2010 MU No No No Nigeria 2010 MU Yes Yes Yes Peru 2010 S No No No West Bank Gaza 2010 MS No No Yes World Bank Engagement at the 2010 - No No Yes State Level: The Cases of Brazil, India, Nigeria, and Russia Bangladesh 2009 MS Yes No Yes Egypt 2009 MS No No No Georgia 2009 MS No No Yes Uganda 2009 MS Yes No Yes Indonesia 2008 MS No No Yes Angola 2007 MU No No Yes Mali 2007 MS No No Yes Madagascar 2006 MU No No No Malawi 2006 U No No No Yemen 2006 MU Yes No Yes Honduras 2005 MU Yes No No