81257 v1 Knowledge-Based Country Programs A n E va l u at i o n o f t h e W o r l d B a n k G r o u p E x p e r i e n c e July 2013 Knowledge-Based Country Programs An Evaluation of World Bank Group Experience Contents ABBREVIATIONS ................................................................................................................................. VI ACKNOWLEDGMENTS ........................................................................................................................ IX OVERVIEW............................................................................................................................................ XI MANAGEMENT RESPONSE ............................................................................................................. XIX MANAGEMENT ACTION RECORD ................................................................................................ XXVII CHAIRPERSON'S SUMMARY: COMMITTEE ON DEVELOPMENT EFFECTIVENESS................ XXXV 1. INTRODUCTION................................................................................................................................. 1 Relative Intensity of Knowledge Services in World Bank Country Programs............................................................ 2 Objectives, Scope, and Methodology of the Evaluation ............................................................................................ 6 2. RELEVANCE OF THE WORLD BANK IN KNOWLEDGE-BASED PARTNERSHIPS .................... 13 Fit of World Bank Knowledge-Based Country Programs with Country Challenges ................................................ 14 Use of World Bank Group Knowledge Services and Stakeholder Engagement ..................................................... 23 Strategic Positioning of the World Bank on Development Knowledge .................................................................... 25 3. TECHNICAL QUALITY OF KNOWLEDGE SERVICES ................................................................... 33 Leveraging the Bank’s Global Expertise ................................................................................................................. 33 Customization of Solutions to Country Context ....................................................................................................... 36 Development of Actionable Recommendations ...................................................................................................... 39 4. RESULTS AND THEIR SUSTAINABILITY ...................................................................................... 43 Results of World Bank Group Knowledge Services ................................................................................................ 44 Factors Associated with the Achievement of Results of World Bank Knowledge Services .................................... 52 Sustainability and Monitoring of Results ................................................................................................................. 56 Synergy Between World Bank and IFC Knowledge Services in Achieving Results ................................................ 60 5. CONCLUSIONS................................................................................................................................ 64 How Can the World Bank Enhance Its Value Proposition in Knowledge-Based Partnerships?.............................. 65 How Can the World Bank Delineate the Scope and Instruments of Knowledge-Based Country Programs? ......... 68 How Can the Bank Group Leverage Its Engagement in Knowledge-Based Country Programs? ........................... 74 Boxes Box 1.1. Definitions, Use of Terms, and Selection of Knowledge Activities............................................. 3 iii Box 1.2. Previous IEG Evaluations of Knowledge Services .................................................................... 8 Box 2.1. Public Knowledge Goods.........................................................................................................17 Box 2.2. Perception of Knowledge Services from Client Surveys ..........................................................29 Box 4.1. Knowledge Service Results in Poverty Reduction and Social Protection: Stylized Findings ...47 Box 4.2. Helping China’s Guangdong Province Tackle Rising Inequality ..............................................48 Box 4.3. Sustainable Knowledge Activity Outcomes Through Capacity Development and Institution Strengthening: Examples from South Africa and China .........................................................................58 Tables Table 1.1. Selected Corporate and Economic Indicators Associated with World Bank Country Programs Categorized According to the Relative Intensity of Knowledge Services (FY05–11) .............................. 5 Table 1.2. IFC Advisory Services and Investment Operations Associated with World Bank Country Programs According to Their Relative Knowledge Service Content (FY05-11) ...................................... 6 Table 1.3. Distribution of Bank Knowledge Services and IFC Advisory Services Across Countries and Thematic Sectors (FY05–11) .................................................................................................................. 9 Table 2.1. Relevance Indicators of Knowledge and Advisory Services Reviewed by IEG (Percent of Reviewed Activities)...............................................................................................................................21 Table 2.2. Outreach of Bank Group Knowledge Services......................................................................25 Table 3.1. Transfer of Global Knowledge in Knowledge Activities and Advisory Services Reviewed by IEG ........................................................................................................................................................34 Table 3.2. Customization to Local Context in Knowledge Activities and Advisory Services Reviewed by IEG ........................................................................................................................................................37 Table 3.3. Development of Recommendations in Knowledge Services and Advisory Services Reviewed by IEG....................................................................................................................................................40 Table 4.1. Achievement of Expected Outcomes of Bank Group Knowledge Services (in % of Activities Reviewed by IEG)..................................................................................................................................45 Table 4.2. Degree of Achievement of Knowledge Services Expected Outcomes in Relation to Knowledge Service Characteristics Evaluated by IEG (in % of Knowledge Services in Each Category of Outcome Achievement) .........................................................................................................................53 Table 4.3. Sustainability of Results and Capacity Building Outcomes of Bank Group Knowledge Services .................................................................................................................................................56 Table 4.4. Sustainability of Knowledge Activity Outcomes in Relation to Knowledge Activity Characteristics Evaluated by IEG (in % of Knowledge Activity by Degree of Outcome Sustainability) ..57 Table 4.5. Results Indicators of World Bank Group Knowledge Services ..............................................59 Table 5.1. Findings, Lessons, and Recommendations ..........................................................................76 Figures Figure 1.1. Lending and Knowledge Activities in World Bank Country Programs (FY98-04 versus FY05- 11) .......................................................................................................................................................... 4 Figure 2.1. Use of World Bank Knowledge Services and IFC Advisory Services (in % of Reviewed Activities) ...............................................................................................................................................23 Figure 2.2. Participation of Government Counterparts in Knowledge Activities (in % of Reviewed Projects) ................................................................................................................................................24 iv CONTENTS Figure 3.1. Use of Local Expertise in World Bank Knowledge Activities (in % of Activities Reviewed by IEG)....................................................................................................................................................... 38 Figure 3.2. Discussion of Capacity Requirements and Administrative Feasibility of Recommendations in World Bank Knowledge Services (in % of Activities Reviewed by IEG) ................................................ 40 Figure 4.1. Achievement of Expected Outcomes of World Bank Knowledge Services by Thematic Area and Focus Countries (in % of Activities Reviewed by IEG) ................................................................... 46 Figure 4.2. Achievement of Expected Outcomes of World Bank Knowledge Activities (in % of Reviewed Projects) ................................................................................................................................................ 51 Figure 4.3. Client Participation in Knowledge Activity Stages in Relation to Degree of Achievement of Knowledge Activity Expected Outcomes (in % of Reviewed Knowledge Activities in Each Category of Outcome Achievement)......................................................................................................................... 54 Appendixes APPENDIX A RANKINGS OF WORLD BANK COUNTRY PROGRAMS ............................................ 81 APPENDIX B CRITERIA FOR COUNTRY PROGRAM AND KNOWLEDGE ACTIVITY SELECTION 85 APPENDIX C CORPORATE AND ECONOMIC INDICATORS ASSOCIATED WITH 48 WORLD BANK PROGRAMS WITH HIGH RELATIVE KNOWLEDGE SERVICE INTENSITY (FY05-11) ......... 88 APPENDIX D APPLICATION OF CRITERIA FOR THE SELECTION OF FOCUS COUNTRY PROGRAMS ......................................................................................................................................... 90 APPENDIX E KNOWLEDGE ACTIVITIES REVIEWED ....................................................................... 92 APPENDIX F EVALUATION TEMPLATE .......................................................................................... 103 APPENDIX G PERCEIVED STRATEGIC POSITIONING OF THE WORLD BANK IN THE FOCUS COUNTRIES ....................................................................................................................................... 109 APPENDIX H EXAMPLES OF SUCCESSFUL KNOWLEDGE ACTIVITIES ..................................... 113 APPENDIX I RESULTS OF WORLD BANK KNOWLEDGE SERVICES IN THE FOCUS COUNTRIES.. ..................................................................................................................................... 117 Evaluation Managers  Caroline Heider Director-General, Evaluation  Nick York Director, IEGCC  Ali Khadr Senior Manager, IEGCC  Aristomene Varoudakis Task Manager  Juan José Fernández-Ansola CoTask Manager  Mauricio Carrizosa CoTask Manager v APPENDIX J PEOPLE INTERVIEWED .............................................................................................. 141 REFERENCES .................................................................................................................................... 149 ENDNOTES......................................................................................................................................... 153 Abbreviations AAA Analytical and advisory activities IDA International Development AIDS Acquired Immune-Deficiency Association Syndrome IDB Inter-American Development Bank ART Antiretroviral treatment IEG Independent Evaluation Group ASEAN Association of Southeast Asian IFC International Finance Corporation Nations IMF International Monetary Fund CAS Country assistance strategy IPRCC International Poverty Reduction CDP Country Development Partnership Center in China CNDP Complementary National Direct JERP Joint Economic Research Program Payment (Kazakhstan) CPAR Country Procurement Assessment JSP Joint Studies Program (Chile) Review KBP Knowledge-based program CPIA Country Policy and Institutional KEIAP Kuwait Education Indicators and Assessment Assessment Project CPS Country partnership strategy LBP Lending-based program CPSCR Country Partnership Strategy M&E Monitoring and evaluation Completion Report MARP Most-at-risk population CSO Civil society organization MSME Micro, small, and medium DAC Development Assistance Committee enterprises DEC Development Economics Vice NCD Non-communicable disease Presidency of the World Bank and OECD Organisation for Economic Co- Chief Economist operation and Development DIPRES Director of Budgets of the Ministry PBB Performance based budgeting of Finance (Chile) PEFA Public Expenditure and Financial ESW Economic and sector work Accountability EU European Union PEMNA Public Expenditure Management FIAS Foreign Investment Advisory Network for Asia Service PEW Public Expenditure Review FIRST Financial Sector Reform and PFM Public financial management Strengthening PIM Public Investment Management FSAP Financial Sector Assessment PISA Program for International Student Program Assessment FY Fiscal Year PPP Purchasing power parity GDP Gross Domestic Product PSD Private Sector Development HIV Human Immunodeficiency Virus QAG Quality Assurance Group IBRD International Bank for RAS Reimbursable Advisory Services Reconstruction and Development RIMSys Results Integration and ICA Investment Climate Assessment Management Systems RMB China Yuan Renminbi vi CONTENTS ROSC Report on the Observance of TA Technical assistance Standards and Codes VCT Voluntary counseling and testing SAR Special administrative region SME Small and medium enterprises vii Acknowledgments This report of the Independent Evaluation Group (IEG) was prepared by a team led by Aristomene Varoudakis and Juan José Fernández-Ansola that included Mauricio Carrizosa, Victor Macias, and Albert Martinez, with contributions from Richard Burcroff, Susan Ann Caceres, Florence Charlier, Ismail Dalla, William Experton, James Hanson, James Lacey, Hernan Levy, Pia Helene Schneider, HjalteSederlof, Xiaolun Sun, Isabelle Tsakok, and Jan Peter Wogart. Analytical support for the review of the World Bank’s portfolio of analytic and advisory activities and the evaluation’s findings was provided by Aida Tapalova and Erkin Yalcin. William Hurlbut edited the report and Aimée Niane provided administrative support. Following Aristomene Varoudakis’s transfer to the Development Economics Vice Presidency in October 2012, Mauricio Carrizosa and Juan José Fernández-Ansola were responsible for revising—in response to management’s and other comments— and finalizing the report. The evaluation benefited from advice and feedback from many persons, in particular the peer reviewers, Shantayanan Devarajan and Shahrokh Fardoust, as well as Martha Ainsworth, Geeta Batra, Hans-Martin Boehmer, Daniela Gressani, Manny Jimenez, Rick Scobey, Mark Sundberg, Marvin Taylor-Dormond, and Christine Wallich. The team gratefully acknowledges the support of management and staff throughout the Bank, and especially in the nine focus countries of the evaluation, who generously shared their time and perspectives with the evaluation team. Special appreciation is extended to many staff and managers, especially in Bank Group country offices, who provided invaluable assistance for the field visits. The evaluation was conducted under the guidance and supervision of Nick York, Director, Country, Corporate, and Global Evaluations, IEG; and Ali Khadr, Senior Manager, Country, Corporate, and Global Evaluations, IEG, under the general direction of Caroline Heider, Director-General, Evaluation. ix Overview Knowledge-Based World Bank Group Country Programs Highlights This evaluation assesses knowledge-based activities in nine country programs selected from 48 knowledge-intensive programs supported by the Bank Group. It identifies the factors in the success or failure of those activities as they contribute to policy making or development outcomes. It also identifies areas of strength for the Bank Group as well as areas of weakness or risk. The evaluation findings are therefore relevant to current Bank Group efforts to strengthen the contribution of knowledge to development. The evaluation was done on economic and sector work and non-lending technical assistance activities selected from a purposive sample of knowledge-intensive country programs. In addition, the evaluation assessed International Finance Corporation Advisory Services for their synergy with the Bank’s analytical and advisory activities. The findings have implications for the Bank Group’s knowledge work, including governance and incentives. The Bank Group remained a strategic partner in the focus countries by providing knowledge services that addressed one or more client needs, which ranged from customized development solutions and capacity development to experience-sharing and innovative ideas. In the sample of countries, the Bank Group was more effective when it worked on specific sectors rather than broad topics, designed tasks to address specific client concerns, customized international best practice to local conditions, generated data to support policy making, and formulated actionable recommendations that fit local administrative and political economy constraints. The Bank Group was less effective when it did not address issues relevant to the client or was unable to follow up consistently with the client on the implementation of advisory activities. Regardless of the level of government that operated as counterpart (central or local), client participation and good monitoring and evaluation systems were key to good results. The evaluation has implications for Bank Group work and for staff incentives. On Bank Group work, it finds a need to emphasize “how to” options rather than diagnostics and “what to do” recommendations; stay engaged and responsive through implementation phases of advisory activities (using programmatic approaches, for example); use local expertise to enhance the impact of advisory activities; design advisory projects with relevant responses to client concerns; and remain engaged in areas that are relevant to a client country’s medium-term development agenda to maintain its capacity to see the big picture and provide multisectoral development solutions. On incentives, an implication is that enhancing the Bank Group’s success rate on providing knowledge services will require staff incentives to be in the knowledge services business. xi OVERVIEW across the 48 knowledge services-intensive Why Evaluate Knowledge Programs Now? programs, the 9 programs selected are fairly The World Bank Group is currently engaged in typical but have a lower lending to knowledge reflection and debate on how to improve the delivery of services ratio; slightly higher share for development support. Part of this debate concerns knowledge services in the country services strengthening the knowledge agenda. The budget; greater average number of knowledge findings of this evaluation are particularly activities because they include large clients relevant because they speak directly to (China and the Russian Federation); slightly questions that the institution is deliberating. higher gross domestic product (GDP) per In particular, they address four key aspects of capita and a significant external current the “science of delivery”: the role of local account surplus because they include China partners or local knowledge hubs; and oil producers; a slightly higher Country consultation with clients and other Policy and Institutional Assessment rating; stakeholders in the process of designing and about the same average volume of World knowledge services; delivery of knowledge on Bank loans. issues that are relevant to the client; and improving the way the Bank Group learns The selection was designed to provide useful from upper-middle-income countries and illustrations of knowledge services effectiveness in the intermediating this knowledge to other selected countries, not to provide a sample for countries. statistical projection to overall advisory activity assistance to those countries or to the The main objective of the evaluation is to learn lessons full set of Bank Group clients. Furthermore, from practices in a focus group of high-income and the selection of countries that included upper-middle-income countries that have knowledge- International Finance Corporation (IFC) based programs with the Bank Group. Over the Advisory Services was designed to illustrate past 15 years, Bank Group country programs complementarities and synergies with the have shifted toward more intensive delivery of Bank in those countries, not to provide a full knowledge services relative to lending, and illustration of IFC Advisory Services this trend is expected to continue. The lessons effectiveness. from this evaluation could help leverage the Bank Group’s global knowledge to meet the Methodology needs of countries that mainly rely on The evaluation categorized Bank Group country knowledge services and are not pressed for programs according to the preponderance of knowledge financing. The nine selected countries are services in program interventions. At one end of the high-income (Kuwait) and upper-middle- spectrum were lending-based programs with a income countries with a high share of predominant role for finance and a relatively knowledge services in their programs, a lower presence of knowledge services. At the diversified economic structure, no or other end were knowledge-based programs moderate Bank lending, and fee-based where knowledge products are at the core of knowledge services. the relationship. The 9 countries selected are fairly representative of The categorization was then used to purposively select countries where the World Bank Group is engaged focus countries that make relatively intensive use of the primarily through knowledge services. They are on Bank’s core knowledge services. The selected the top half of a ranking of relative countries were Bulgaria, Chile, China, preponderance of knowledge services, with 48 Kazakhstan, Kuwait, Malaysia, Russian countries fitting a definition of knowledge- Federation, South Africa, and Thailand. To based programs. Compared with the average xii OVERVIEW probe the synergy of the Bank’s knowledge development outcomes were already in services with IFC Advisory Services, the evidence, such as a report on inequality in evaluation also examined those services in the China. focus countries when a government entity was the recipient. Achieving Outcomes: The Main Success Factors The selection of economic and sector work (ESW), In the focus countries, intended outcomes were fully technical assistance (TA), and IFC Advisory Services achieved or likely to be achieved in 47 percent of the paid attention to links to the strategic priorities in the knowledge activities reviewed and partly achieved in country partnership strategy of the focus country. The another 37 percent. The frequency of outcome knowledge services sampling relied on achievement was broadly equivalent for Bank consultations with country management units. ESW and TA—the Independent Evaluation The sample includes World Bank knowledge Group (IEG) did not find a significant services and IFC Advisory Services delivered difference in outcomes between the various over FY05–11. IFC Advisory Services were models of knowledge service delivery. reviewed in China, Kazakhstan, the Russian Outcomes of IFC’s Advisory Services were Federation, and South Africa. The sample achieved in about 38 percent of the small consists of 266 Bank analytical and advisory sample of projects reviewed. The achievement activities (AAA) and 34 IFC Advisory Services of outcomes of knowledge services in the nine out of a total of 751 Bank AAA products and focus countries was comparable to that of 185 IFC advisory service projects (to both Bank Group lending operations across government and private sector recipients) regions. The Bank Group was more effective delivered over FY05-11. Products with similar when it worked on specific sectors rather than thematic focus were grouped in clusters of broad topics; designed tasks to address knowledge activities. Thus, the number of specific client concerns; customized activities reviewed was 196 for Bank AAA and international best practice to local conditions; 32 for IFC Advisory Services. generated data to support policy making; and The selected knowledge activities in the nine focus formulated actionable recommendations that countries were assessed against four criteria: fit local administrative and political economy relevance of the knowledge activities to the constraints. Regardless of the level of priority needs of the recipients and the key government that operated as counterpart development goals of the client country; (central or local), client participation and good technical quality of the activities in leveraging monitoring and evaluation (M&E) systems the Bank’s global knowledge and conveying were key to good results. relevant and customized expertise to Outcomes were more likely to be achieved when the recipients; results achieved; and sustainability knowledge services focused on specific sectors, such as of results. agriculture and rural development, education The assessment of outcomes was based on the feedback and health, and the financial sector. Reaching obtained during country visits and through desk outcomes proved more difficult in broader reviews. The assessed progress ranged from thematic areas, encompassing an ambitious tasks that had little or no impact on policies, reform agenda, or when the achievement of such as Investment Climate Assessments in results required multisector efforts, such as Thailand, to tasks with recommendations that private sector development, economic policy, were being implemented, but with no visible and public sector governance. In such impact on policies, such as a report on complex areas, knowledge service results student loans in Chile, and to tasks where often suffered when new legislation was xiii OVERVIEW necessary before the recommended reforms gradually implemented with concrete results could be implemented. For example, in in declining inequality. In Thailand, contrary Kuwait, interactions between parliament and to other development agencies, counterparts the executive complicated the passing of laws see the Bank as having the capacity to in several areas of Bank TA, such as properly customize international best practice procurement, public finance, civil service, to the Thai context because of its knowledge freedom of information, and anticorruption. of local institutions that comes mainly from the expertise of staff in the Regional Country Knowledge services used in the design of lending Office in Bangkok. The Thai report The operations were more likely to succeed than Economics of Effective AIDS Treatment is a good freestanding knowledge services. Although lending example of customization to country context. was limited in most of the focus countries, it In instances where the Bank did not fully remained a powerful driver of results for the address issues relevant to the client, results of Bank’s knowledge services as at least partial knowledge-based activities tended to be achievement of expected outcomes of Bank poorer. Knowledge services that lagged in the knowledge services was observed more achievement of outcomes were also weak in frequently when knowledge services were conveying international best practice, used for the design of lending operations. providing relevant examples, producing new Possible explanations are that in this instance evidence and data useful for policy making, the Bank has more leverage than with formulating actionable recommendations, and freestanding knowledge services, and also that discussing the capacity requirements and the knowledge services by definition are administrative feasibility of implementing supporting a program that is expected to be recommendations. implemented. Freestanding knowledge services many times contributed to policy Knowledge services with fully or partly achieved discussions where the authorities had not yet outcomes were more likely to use local expertise. Use taken a position. of local experts and counterpart participation appear to help modify global best practices to The achievement of outcomes was not correlated with fit local conditions, formulate financing arrangements for knowledge services—Bank recommendations that account for capacity or client—probably owing to the high constraints, and improve stakeholder relevance to the client of Bank knowledge ownership of findings and suggested actions. services in most of the focus countries. Other Client participation in the various stages of factors—related to the relevance of design, knowledge services also was associated with quality, timeliness, client participation, and use the achievement of results. Moreover, of local expertise—were more closely knowledge services that achieved results have associated with achievement of results than more often contributed to strengthening source of financing. institutions as well as analytical and policy formulation capacity of recipients. The China Knowledge services requested by the client and designed Preparation of Capital Market Development report specifically to achieve client objectives were more likely is an example of detailed coverage of the to achieve outcomes than knowledge services of a more institutional and policy context and reliance generic character. For example, in China there is on a local team of experts to draft the report evidence that the recommendations of the in Chinese using existing data. The report report Reducing Inequality for Shared Growth in contributed to capacity building at the China: Strategy and Policy Options for Guangdong Research Center of China Securities Province, a high-profile study conducted jointly Regulatory Commission and to raising its with the provincial authorities, are being xiv OVERVIEW profile and role as the capital markets helped build capacity either through training, regulator. networking, or access to international best practice. Public knowledge goods typically The outcomes of about 75 percent of the Bank consisted of Bank-funded reports available to knowledge services and the IFC Advisory Services a broad audience primarily for disseminating were likely to be at least partly sustained. That is, analyses of developments (such as Economic knowledge services were likely to have (at Monitoring Reports) or particular sectors or least partial) lasting impacts on policies, issues (such as investment climate assessments capacity, or institutions. and financial sector assessments). The majority of these knowledge products conveyed The Bank’s main strength, which reflected international best practice and relevant examples, recommendations from previous IEG knowledge generated new evidence to inform policy making, and services evaluations, was its ability to fulfill in a timely formulated actionable recommendations consistent with manner client requests for state-of-the-art advice. the findings. Sustainability of outcomes was Clients found most value in the Bank’s ability more often observed when knowledge to address relevant developmental issues, services were complemented by other World convey international best practice, act as a Bank activities (lending, other ESW, or trusted knowledge broker, customize complementary TA). In the majority of cases knowledge to the local context, and take a where sustainability of outcomes was likely, pragmatic approach to important issues that knowledge services contributed to required multisectoral development solutions. strengthening institutions or the analytical and Counterparts interviewed by IEG policy formulation capacity of recipients. acknowledged four key strengths: ability to About 60 percent of Bank knowledge services benchmark against international best practice contributed at least partly to developing or through cross-country comparisons, strengthening institutions—with a much reputation as an independent and credible lower frequency in the case of IFC Advisory broker of knowledge with a partnership Services. Similarly, a large majority of Bank approach, knowledge of the local context and knowledge services and a significant part of capacity to customize international best IFC Advisory Services contributed to practice solutions, and capacity to see the big strengthening analytical or policy formulation picture and analyze cross-sectoral issues capacity of recipients. important for development. Timely delivery of knowledge services to affect important Areas of Bank Strength decisions was essential to achieve the The Bank remained relevant and a strategic partner expected outcomes. in the focus countries by providing knowledge services that addressed one or more client needs. Customized Another key strength was linked to its role as development solutions filled a knowledge gap “knowledge connector.” The Bank’s convening in an area where counterparts needed timely power was often used to mobilize top and actionable recommendations to develop a international experts for brainstorming strategy or take action. In experience-sharing sessions and seminars with high-level and innovative ideas on issues where government officials, or for TA and working counterparts had not yet taken a position, the sessions with government agencies. There are Bank functioned as a sounding board or some good examples where the Bank’s connected counterparts to cutting-edge knowledge activities facilitated South-South international expertise. Capacity development exchanges and policy dialogue in the focus was provided in the form of knowledge that countries. The Bank has used mostly its informal networks, through the task team xv OVERVIEW leaders and network management, to convey reinforced good joint work. Another factor knowledge acquired in Chile to other was the existence of core ESW—such as countries in the Latin American region and investment climate assessments and financial elsewhere. In Kazakhstan, some government sector assessments—which underpinned the agencies have already shared their experiences strategy and helped identify priorities for with other countries in the region. But more improving the investment climate and can be done as the Bank’s geographic, developing the financial sector. thematic, and organizational fragmentation prevents the full potential of such exchanges Areas of Bank Weakness or Risk from being realized. In China, for example, Poor achievement of outcomes was associated with while the World Bank Institute’s technical weaknesses in relevance of design, quality, timeliness of assistance work has focused on catalyzing delivery, or client participation, and little use of local lessons for other development countries, it expertise. Knowledge services that lagged in the appears that the Bank has not fully exploited achievement of outcomes were also weak in the potential of this mutual lending conveying international best practice, opportunity. The Russian Federation could providing relevant examples, producing new benefit from the extensive work the Bank has evidence and data useful for policy making, done in China on regional approaches to formulating actionable recommendations, and investment promotion, but lessons from this discussing the capacity requirements and experience have not been transmitted to the administrative feasibility of implementing Russian Federation. recommendations. Where the Bank was less able to address issues relevant to the client it Bank knowledge services and IFC Advisory Services also tended to achieve poorer results. The lack generally complemented one another in contributing to of timely delivery of knowledge services to results, despite some gaps. In a few cases there affect important decisions—not a prevalent were well-defined programs of joint World problem in the sample of countries—also was Bank and IFC knowledge activities. For associated with poorer outcomes. example in South Africa, to ensure synergy and coordination, IFC used experienced Bank The Bank’s ability to customize knowledge services to staff to manage projects on enterprise tax the local context and to deliver multisectoral solutions burden and compliance. Other examples are is at risk of eroding where country knowledge is too in the investment climate area, where World shallow or too narrow. This risk arises mainly Bank–IFC collaboration allowed both to when the Bank works through Reimbursable contribute to policy dialogue and reform Advisory Services (RAS) and does not activities. But overall the experience with maintain a local presence. The Bank’s coordination between World Bank and IFC strengths may also be challenged by its was mixed. Coordination can be improved, increasing tendency to deliver knowledge for example, by establishing more systematic services through the “consultant firm model,” mechanisms for inclusion of one institution in with insufficient follow up and emphasis on the other’s review processes, especially at the important issues for the medium-term concept and design stage. A key factor development agenda. There is a tension supporting the synergy of the two institutions between the Bank as a development agency— in achieving results was the quality of the focusing on important medium-term results framework in the country partnership development issues—and the Bank providing strategy (CPS). The articulation of strategic specific solutions to narrower problems outcomes and the clarity of links between the suggested by the main counterpart in the Bank Group’s programs, projects, and country—generally a unit within the Ministry instruments with expected outcomes xvi OVERVIEW of Finance. To the extent that such a unit practical know-how and enable customization mars the Bank’s engagement in relevant of global practice. broader development issues, the Bank’s overall mission in the country could be Stay engaged and responsive in the impaired. This tension on who is the client implementation phases of advisory needs to be resolved case-by-case using activities through instruments that help substantial diplomatic tact but emphasizing clients translate recommendations from the Bank’s broader development mandate. sound analysis into actions that fit local political and administrative constraints. Monitoring of Bank knowledge services results was Use of programmatic approaches was weak—both for individual activities and for country important to achieve outcomes. IEG programs. In only 17 percent of the knowledge recommends the Bank design programmatic activities assessed was there at least a partial knowledge services in a number of well- results framework in the CPS, allowing a defined thematic areas (such as public tracking of the contribution of the activity to financial management) that build on initial the broader development outcomes sought by work to support implementation phases, the CPS. Similarly, only 23 percent of the including engagement of a broad range of knowledge services included, at least partly, stakeholders to help disseminate the reform result indicators to track the achievement of agenda and maintain the focus on key policy the activity’s outcomes. In contrast to Bank issues in the public domain. knowledge services, the great majority of IFC Advisory Services reviewed by IEG were at Where applicable, ensure links among least partly equipped with results indicators to Bank ESW, non-lending TA, and projects trace achievement of outcomes. Monitoring to help sustain results. When knowledge of capacity development outcomes and lesson services were complemented with lending, learning were, on average, weak for Bank results were more likely to be sustained. IEG knowledge services, but less so in the case of recommends the Bank design CPSs with knowledge services with outcomes likely to be closer links between knowledge services and achieved. lending, including programmatic series deploying both instruments to support the Implications for Bank Group Work paths from consideration of policy options to implementation of the approaches selected. Emphasize the “how to” options, as opposed to the diagnostics and the “what Clarify the political economy of reform to do” recommendations, to enhance and use local expertise to enhance the client ability to own final policy decisions, impact of knowledge services. Local action plans, or strategies. In general, tasks partners and Bank Group hubs can be critical that achieved results provided actionable in conveying relevant country context recommendations more often than those that considerations. Most of the tasks reviewed did not achieve results. IEG recommends the referred to the local policy context in varying Bank give staff more time to interact with levels of detail. Those that achieved results clients and local partners and knowledge probed more deeply into the country context hubs, including through adequate field and used local expertise more often than presence. Moreover, use analytical resources those that did not achieve results. IEG intensively to ensure that high-quality research recommends the Bank involve local experts, underpins recommendations, and deploy partners, and local knowledge hubs more high-level consultant expertise able to provide extensively in knowledge services to help better understand the political economy of xvii OVERVIEW reform in the country where advice is sought, outcomes and their linkages with the bridge the gap between international good programs and instruments of both practices and local conditions, enhance the institutions. Complement this with formal applicability of the recommendations, and mechanisms of including each institution in build the local capacity to achieve longer-term the other’s review processes and better impact. coordination in the field. Pay attention to the quality and relevance Remain engaged in areas that are relevant of knowledge services. This is essential for to a client country’s medium-term obtaining results—regardless of the form of development agenda to maintain the financing of knowledge services—and for capacity to see the big picture and provide shaping learning under the Bank’s “science of multisectoral development solutions. This delivery.” Staff needs to take multiple actions capacity has been a strong point, generally to achieve results: design projects with valued by clients, of the Bank’s knowledge relevant responses to client concerns; services. Delivery of knowledge services customize international best practice to local through a “consultant firm model,” which conditions, including capacity constraints; reflects a drive to accommodate multiple and generate data to support evidence-based unforeseen needs, often results in policy making; formulate actionable fragmentation of Reimbursable Advisory recommendations that fit local administrative Services (RAS) programs—for example, by and political economy constraints; and deliver dropping tasks linked to medium-term products in time to influence key decisions. objectives to accommodate shorter-term IEG recommends the Bank consult broadly needs—and may dilute the focus on with the client and other stakeholders on the important medium-term development issues. issues to be addressed, deploy highly For countries where most of the activities are experienced staff with global perspective and knowledge-based and the Bank Group does ability to deliver knowledge services on time, not have a CPS, IEG recommends that the and adhere to the mandatory knowledge Bank Group prepare CPSs (which do not services quality assurance process. The Bank have to follow a burdensome consultation Group should encourage emerging process) to provide guidance on engagement “knowledge hubs” to follow approaches along objectives and avoid fragmentation of these lines. knowledge services away from evolving development priorities. Furthermore, the Strengthen synergies between World Bank Bank may consider using instruments (such as knowledge services and IFC Advisory high-level brainstorming, conferences, and Services projects to improve results. ESW, including periodic Economic Linking Bank and IFC activities also helped Monitoring Reports) and committing the achieve results. But experience with Bank– necessary resources to identify, follow up, and IFC coordination has been mixed, with quality sustain emphasis on issues that are important of the CPS results framework and the for medium-term development. existence of core ESW among the factors that influenced the degree to which World Bank Undertake broad-based consultations and and IFC knowledge services had synergy. IEG dissemination, acknowledging the public recommends the Bank conduct core good function of Bank knowledge services knowledge services for private and financial while paying attention to local sector development and develop joint Bank– circumstances. Client participation in the IFC programs and projects within the CPS different stages of knowledge services appears results frameworks that articulate the to be closely associated with success in xviii OVERVIEW achieving expected knowledge service more systematic mechanisms to include each outcomes. IEG recommends the Bank organization in the other’s review processes. broaden the participation of various An additional challenge—given the vision of a stakeholders into knowledge-based country Bank Group for the whole world—will be to programs (for example, by opening up bring in the knowledge acquired from discussions or focus groups with local experts knowledge-based partnerships to lower- and civil society organizations) and make income countries. Bank studies more widely accessible (for example, by recognizing the public good Strengthen Bank learning from upper- component in knowledge products and middle-income countries and the sharing a portion of the knowledge services intermediation of this knowledge to other cost with the client on the condition of its countries. There were ample opportunities disclosure). for learning from development experiences in the focus countries (for example, on the Monitor closely implementation and development trajectory from a low-income to results to track progress toward mutually an upper-middle-income economy in Malaysia agreed outcomes and mitigate the risk of or the extensive work the Bank has done in fragmentation and loss of strategic focus China on regional approaches to investment that are intrinsic to RAS. Bank knowledge promotion). IEG recommends the Bank services were not monitored and evaluated enhance exchanges of knowledge within the consistently in the sample of countries. Where Bank through communities of practice and M&E was better, knowledge services results outside the Bank through networks of were more likely to be achieved, probably practitioners or knowledge hubs; enhance the reflecting a link between M&E, knowledge links of the Bank’s regional chief economists service quality, and impact. IEG recommends with regional institutions that can play a role the Bank use—and continuously improve— in sharing the Bank’s analytical work; ease the implementation and results monitoring confidentiality of knowledge activities systems that would track progress toward conducted through RAS; and leverage the achieving the outcomes in the results technical capacity developed by upper-middle- framework of the CPS and that knowledge income clients to other countries (for activities be more tightly linked with CPS example, partner with Thai institutions to milestones and outcome indicators. A “circle bring in the experience of Thailand’s built of continuous quality improvement” for capacity in banking, payments system, and M&E is critical for shaping the “science of financial markets to other countries that may delivery” that the Bank is presently intent need it). upon and to help improve M&E at the country level. Continue to use RAS to expand the feasible set of Bank services, ensure the Reinforce knowledge services governance sustainability of the Bank’s business and partnerships to help enhance results. model in knowledge-based country Governance will benefit from management programs, and generate new knowledge leadership to encourage knowledge services that the Bank can then intermediate to and develop stronger M&E for knowledge lower-income countries. Although the services. World Bank–IFC coordination can relevance of RAS is strengthened by client be strengthened by developing high-quality demand and financial commitment, results do CPS results frameworks that clearly articulate not appear significantly different from those links between outcomes and World Bank of knowledge services funded by the Bank’s Group advisory activities, and by establishing own resources. Other fundamental success xix OVERVIEW factors—related to the relevance of design, recognizes that knowledge is the foundation quality, timeliness, client participation, use of of development, and that the Bank should do local expertise—are more closely associated more in this regard. The incentives for staff to with the achievement of results. IEG engage in knowledge activities need to be as recommends the Bank move decisively strong as those for being part of lending toward RAS in knowledge-based programs to operations. At a minimum the Bank should sustain this business line, while clarifying the review the incentive system for staff, ensure types of knowledge services that come close that knowledge contributions are recognized to “public knowledge goods” (Bank–funded for career advancement; make bringing reports targeted to a broad audience to knowledge to countries a visible priority; and disseminate analyses of developments or ensure that personal reputations of staff are particular sectors or issues), as opposed to enhanced by knowledge contributions. This those that serve specific needs of evaluation suggests the need to deploy part of counterparts. RAS could be offered to the best staff in the institution to knowledge institutions that can cover the full cost of the services. Those dimensions that got in the way Bank’s services, with cost-sharing of of achieving results in some instances—poor knowledge activities that are not pure public design relevance; weaknesses in conveying knowledge goods. The cost-sharing would international best practice, providing relevant recognize the relevance of the activities for examples, producing new evidence and data other countries and provide for their wider useful for policy making, formulating disclosure or dissemination by the recipient or actionable recommendations, or discussing for equalization of access to the Bank’s the capacity requirements and administrative knowledge services among subnational clients feasibility of implementing with varying capacity to pay for the service. recommendations—should be addressed by For countries involved in cost-sharing—Chile allocating high-caliber staff to the provision of and Kazakhstan, for example—the disclosure knowledge services. Such staff would also be or dissemination to other clients would give able to address issues relevant to the client— them a sense of contributing to the global one of the key factors in achieving successful public knowledge goods agenda. Where there knowledge service outcomes. Staff incentives is full cost recovery, it could include the cost for knowledge activities will need to be of a client survey with the aim of informing balanced with rewards for engaging in other both the Bank and the client about the important priorities such as lending and work relevance, quality, use, and results achieved or in fragile states. likely to be achieved. Implications for Staff Incentives Enhancing the Bank Group’s success rate on providing knowledge services will require that the highest-caliber staff have incentives to be in the knowledge services business. There is a perception among Bank staff that lending experience is essential for promotion. At the same time, the world xx Management Response World Bank Group management thanks the Independent Evaluation Group (IEG) for undertaking this valuable evaluation of knowledge-based programs in eight upper-middle- income countries and one high-income country. Management welcomes the opportunity to comment on the draft evaluation report. This is timely in view of the Bank Group’s increased focus on measuring results in its knowledge activities. The next section summarizes World Bank Group response to the evaluation. Management’s specific response to IEG’s recommendations, with which it generally agrees, is noted in the attached draft Management Action Record (MAR) matrix. World Bank Management Response This is a well-written report with useful lessons as the Bank moves forward on the knowledge agenda at the country and global levels. The analysis tackles some important issues ranging from staff incentives to internal and external factors of success and the report skillfully captures and synthesizes the key messages. Management welcomes the thrust of many of the overall findings, which, in general, endorse much of the Bank's approach to knowledge-based work. The fact that our knowledge services (in these "knowledge-based country programs") are judged to be relevant and generally well regarded by our clients is encouraging. In particular, the report notes that 84 percent of the Knowledge and Advisory Services (with a focus on the Bank’s Economic and Sector Work/Technical Assistance or ESW/TA) had fully achieved or partially achieved objectives. Likewise, it is encouraging that "state-of-the-art advice" and "knowledge connector" are seen as Bank strengths. Appendix G of the paper also notes that for countries such as Chile and Bulgaria, the Bank compares favorably to other international institutions, think tanks, and global consulting firms regarding quality, relevance, and timeliness of its knowledge. All of this points to a significant vote of client confidence in the Bank. Management is in broad agreement with the recommendations made in the report (and summarized in the attached matrix). Implementation of many of the recommendations are already underway. However, we also note that the scope of the evaluation does not fully represent the full range of the Bank’s knowledge services; it limits the definition of knowledge-based programs largely to upper-middle-income countries (UMICs) and excludes the vast knowledge base that underpins programs in International Development Association (IDA) and fragile countries; and it does not reflect actions taken to strengthen the results framework, adopted in FY11 and that are summarized in Box 1. xxi MANAGEMENT RESPONSE Box 1: Progress in Developing a Results Framework for Knowledge since FY11  The following actions have been taken to strengthen results since the IEG evaluation:  ESW/TA intermediate outcome indicators have been updated to be more outcome- oriented.  The task team self-assessment process has been strengthened and hardwired into the Bank’s management information system.  Tools for managers to facilitate greater oversight of quality are now captured in the Ops Portal.  Guidelines have been developed for reimbursable advisory services (RAS) that facilitates consistency in processing steps and quality assurance across the Bank.  Client feedback instruments have been put in place for ESW/TA/TE and KP that has been used with the self-assessment to analyze results.  Programmatic ESW/TA has been adopted to strengthen the strategic orientation of the Bank’s knowledge work. This approach is being expanded to accommodate other knowledge products. It is important to note that the study does not fully represent the full scope of the Bank’s knowledge work in client countries. The work does not acknowledge the fact that all of the Bank’s country programs are knowledge-based or have a strong knowledge content. What distinguishes the set of activities covered by this evaluation is not so much the presence of knowledge as the absence of lending. In some of our country programs which do include borrowing, the amount of knowledge work is much greater than in some of the countries covered by this study. We would suggest IEG consider modifying the title of this evaluation as it currently seems to imply it covered all of the Bank’s knowledge work, when it is actually limited to nine country programs in middle- and high-income countries. It is also important to note that the report focuses mainly on ESW/TA, which comprises about 60 percent of the Bank’s core knowledge services. While ESW/TA is a critical element of the Bank’s work, management would appreciate an understanding of whether and how IEG proposes to evaluate the remaining core knowledge services, which represent nearly $300 million in expenditures each year. The report also notes key areas for improvement related to: relevance and technical quality; strategic orientation and sustained follow-up of its knowledge work (particularly in cases where RAS is the primary form of engagement); results focus through strengthening the monitoring and evaluation framework for knowledge; and incentives for knowledge work. The rest of this section outlines the main recommendations and the Bank’s response. The report recommends that the Bank ensure relevance and technical quality through the customization of international evidence to formulate policy options that fit local conditions and generate data to support policy making. This is also linked to the recommendation that the World Bank Group clarify the political economy of reform and use local expertise to xxii MANAGEMENT RESPONSE enhance the impact of knowledge work. It notes that those knowledge activities that achieved results probed more deeply into the country context and used local expertise more often than those that did not achieve results. The use of local experts, partners, and local think tanks help to better understand the political economy of reform in the country where advice is sought, bridge the gap between international good practices and local conditions, enhance the relevance and applicability of the recommendations, and build the local capacity to achieve longer-term impact. The report recommends that the Bank Group be more strategic, including putting in place mechanisms for follow-up and sustained engagement. This requires that the Bank stay engaged and responsive in the implementation phases of advisory activities through appropriate instruments that help clients translate recommendations from sound analysis into actions that fit local political and administrative constraints. The report makes the link between the use of programmatic approaches as a tool for sustained engagement and the achievement of high-impact or greater results. The Bank has developed and operationalized a framework (including an approach to monitoring and evaluation) and the allocation of resources that enables use of an explicit programmatic approach (PA). This tool became available on the Operations Portal in FY13. Reimbursable Advisory Services. The report also notes the need to manage RASs more strategically. A new tool for managing RAS became available on the Operations Portal in FY13. Discussions with Bank management and the Board about the RAS strategy are ongoing. A FY12 review of RAS (at the time called Fee-Based Services or Reimbursable Technical Assistance) was completed in October of 2012 (“RAS Review”). The review showed that the nature and amounts of RAS are growing and rapidly evolving. Given the importance of RAS for UMICs, management particularly welcomes the report’s assessment of this aspect of the Bank’s knowledge work. IEG findings raised issues around confidentiality which may be hampering the extent of broad-based engagement and dissemination of the Bank’s knowledge work in client countries. The report also highlighted the tensions around the “consultancy firm” model that is potentially limiting the Bank’s ability to engage strategically and emphasize broader development issues in a given country. Despite these challenges, IEG recommends the Bank move decisively towards RAS in knowledge-based programs to sustain this business line. Recommendations also highlighted the need to have the right incentives in place for RAS knowledge work to maintain a focus on development outcomes. The Bank has been grappling with the above issues (most of which were also addressed in the 2011 RAS review). Management has the following comments related to confidentiality, quality assurance, and the “consultancy firm” model and associated risks. On confidentiality: The report proposes that the Bank seek to relax some of the confidentiality surrounding RAS. The RAS Review (2012) did look at this issue and the confidentiality guidance for RAS engagements was consequently amended to allow for more openness, with policy guidance to seek client consent for disclosure on a regular basis. The xxiii MANAGEMENT RESPONSE changes were constrained by the Access to Information Policy (AIP), which treats RAS differently from other Bank operations, requiring client consent for disclosure. On quality assurance: The RAS Review also looked at quality assurance issues and identified this as a priority area for improving the program. Quality assurance for RAS is, in theory, analogous to that of Bank knowledge operations. The modalities for putting this in practice have now been fully elaborated and adopted. The new three pronged approach to monitoring results is also being applied to RAS. Operational Policy and Country Services (OPCS) (with others) has undertaken a number of steps to ensure this approach is tailored to the peculiarities of RAS, including through integrating RAS into the Operations Portal — i.e., the Bank’s management information system for knowledge — and through detailed guidance on the application of operational policies to the RAS program. The report raises a number of issues around the ‘consultancy firm’ model. In this connection, management would like to point out that, as an institution, the Bank has never taken the policy position that it can or should act as a consultancy firm. This position underlies a number of the features of the RAS program, including the provision that the Bank does not engage in competitive processes for RAS work. Similarly, it informs a number of the provisions in the model RAS legal agreement on intellectual property and liability, among other things. The issue was considered both in 2008 and during the FY12 RAS Review of the program, since the Bank is often under pressure from clients to conform more closely to private sector practices. On both occasions, the consensus was that the Bank should not follow a private sector business model, as it would be inconsistent with the Bank's mandate and status as an international financial institution. Having noted this, management recognizes the tensions and incentives in a budget constrained environment toward the “consultancy firm” model. Recently adopted policies and guidance on when to undertake a RAS and the related rules and standards around quality assurance, legal agreements, and risks are designed to manage these tensions and related risks. The report recommends that the Bank Group improve the results measurement system for knowledge services. The weakness in monitoring ESW/TA over the period of the review (FY2005-11) is rightly noted in the report. Since FY11, the Bank has adopted a new three- pronged approach to results and measurement for knowledge (that applies to all client-facing knowledge including RAS). The approach involves: (i) strengthening the self-assessments; and this will build on recommendations from IEG’s ongoing review of ESW/TA in the Operations Portal; (ii) implementing client feedback instruments to elicit client assessment of the usefulness and relevance of our knowledge products; and (iii) ex-post review of Bank assessments by IEG (or other external evaluators). This approach is now under implementation for all external client-facing products and for internal knowledge products. Work is ongoing through systems improvement to reflect both stronger results monitoring and streamlined accountability across all products (lending and knowledge services). In addition, the framework encourages staff to think in terms of a theory of change with clear monitoring indicators to help task teams structure their knowledge activities. This could be applied to specific knowledge products (including ESW/TA) that are linked to the CPS results frameworks. The framework also requires teams to assess the risks associated with the knowledge activity achieving its intended objective, including an assessment of the relevant xxiv MANAGEMENT RESPONSE stakeholders and change agents and actions to mitigate these risks. The aim is to have the resulting plans for risk mitigation, supported by stakeholder analysis. These would reinforce the need to engage local actors to better manage risks associated with the political economy of the knowledge activity or engagement. The report highlights the need to strengthen incentives for knowledge work ensuring that the appropriate resources and staff are deployed. The report recommends adoption of a financial and budget framework that balances managerial incentives for lending and knowledge activities. The Bank has already initiated actions to improve management oversight to signal senior management’s increased attention to results-oriented knowledge work. In addition to the actions outlined in the paragraphs above, the Bank has adopted an approach to link client feedback to management through the memoranda of understanding (MoUs) of regional and network vice presidents. These actions coupled with the broader focus on results and client orientation will help support the appropriate incentives for knowledge work. IEG’s Recommendations Management agrees with the key recommendations on ways to strengthen quality, client, and results orientation, including incentives for knowledge work. Specifically, the report states the Bank should focus on “how to” options rather than diagnostics and “what to do” recommendations; stay engaged and responsive through implementation phases of advisory activities (including through programmatic approaches); use local expertise to enhance the impact of advisory activities; design advisory projects with relevant responses to client concerns; and ensure the Bank’s engagement is strategic with a focus on contributions to outcomes that are relevant to a client country’s medium-term development agenda. Since the period under review in the report, management has initiated a number of actions to strengthen accountability and results orientation. Specifically, management notes that it has already been actively implementing many activities recommended by IEG, as noted in the attached MAR matrix. The allocation of resources to support knowledge products is already driven by the country program's expected results via the country assistance strategy (CAS)/CPS for result processes. Client and Bank teams determine the development outcomes and the Bank’s activities needed to support the outcomes. Resources are then allocated to support the required activities. In many cases these are stand-alone knowledge products that are unrelated to lending. While management is committed to align incentives to achieve excellence in knowledge services, and, in general, to achieve improved development results, it cannot commit to specific budget allocation principles at the time when the entire budget process is being reviewed to align it to the new World Bank Group strategy under development. Management sees IEG playing a key role in strengthening the monitoring and ex-post evaluation of the Bank Group’s knowledge services. Specifically, management looks forward to receiving the results of IEG’s ongoing review of the Bank’s information management system for knowledge to ensure it will adequately support decision making and ex-post evaluation. Results of the review will be used to further strengthen existing systems and processes for capturing and assessing knowledge. xxv MANAGEMENT RESPONSE International Finance Corporation Management Response International Finance Corporation (IFC) management welcomes IEG’s recognition of the many reforms IFC has introduced in recent years to strengthen the impact and effectiveness of our Advisory Services projects. Since most of the limited number of IFC projects reviewed in this report were designed prior to those reforms, care should be taken in interpreting them as representative of IFC’s current approach. xxvi Management Action Record IEG Findings & IEG Recommendations Acceptance by Management Response Conclusions Management of Recommendatio n Relevance and technical Ensuring relevance and WB: Agree WB: We agree with the need to ensure relevance and quality of results were key technical quality: technical quality. As noted in the findings, we see the most drivers for achievement of Customize international effective way to do this is to work closely with the client to results of knowledge evidence to formulate policy develop actionable recommendations that address the key services. Key dimensions options that fit local development objectives/goals. Implementation of this included designing tasks that conditions and generate data recommendation is already underway. responded to client concerns; to support policy making. customizing international Deploy highly experienced Various evaluations at the Bank show that the Bank’s best practice to local staff with global perspective, knowledge is rated well in terms of technical quality (i.e., conditions; generating data and strengthen quality IEG’s 2008 evaluation of ESW/TA; the most recent client to support policy making; assurance process for surveys both the product based Client Feedback Instrument and delivering products in a knowledge services. or CFI, and the Country Opinion Survey Program or timely way to influence key COSP). These same instruments highlight the need for decisions. Moreover, client stronger client engagement. The Bank is strengthening the participation in the different monitoring and reporting system that tracks partnerships stages of knowledge and client engagement for knowledge. activities appears to be closely associated with Quality assurance continues to be strengthened and a success in achieving number of changes have been implemented since this report expected knowledge service was completed: the results framework for ESW/TA was outcomes, and tasks that strengthened to be more outcome-oriented; the self- achieved results provided assessment process was strengthened and hardwired into the actionable recommendations Bank’s management information system with tools xxvii MANAGEMENT ACTION RECORD IEG Findings & IEG Recommendations Acceptance by Management Response Conclusions Management of Recommendatio n more often than those that managers can use for greater oversight of quality; more did not achieve results. systematic guidelines for RAS that facilitates consistency in processing steps and quality assurance across the Bank are in place; finally, the Bank’s improved system for quality assurance makes provision for capturing lessons learned in searchable fields that will enable greater learning and data to support policy advice. Most tasks referred to the Making use of political WB: Agree WB: Implementation of this recommendation is already local policy context. Those economy analysis and local underway. We agree that consideration of political economy that achieved results probed knowledge: Involve local issues are critical and we have initiated consultation around more deeply into the local experts, partners and local how to strengthen this aspect of our work. Already, as part context, used local expertise think tanks extensively in of the recent improvements to strengthen client and results more often than those that knowledge services to help orientation, we are requiring task teams to explicitly outline did not achieve results, and understand better the the risks they see to the achievement of targeted objectives formulated actionable political economy of reform, for a knowledge activity, and client adoption of recommendations to fit local bridge the gap between recommendations. Teams will be asked to identify risks administrative and political international good practices related to design of the knowledge activity/engagement, in economy constraints. and local conditions, enhance line with the approach used for lending services, and within Understanding the political the applicability of the the strategic context of the Bank’s country engagement economy of reform and using recommendations, and build through the CAS/CPS. Guidance and learning material to local expertise can enhance local capacity to achieve support staff are being developed. Going forward, a the impact of the Bank’s longer-term impact. Stay question will be included in the CFI to help monitor the use knowledge services. Local engaged with client, know of local institutions and civil society. partners or hubs can also the local context, and play a critical role in strengthen implementation The Bank will also strengthen its approach to improved conveying relevant country by mobilizing international learning from operations (based on case studies and context considerations. expertise in a timely fashion. toolkits). Steps will also be taken to promote shorter xxviii MANAGEMENT ACTION RECORD IEG Findings & IEG Recommendations Acceptance by Management Response Conclusions Management of Recommendatio n Involve local partners in learning loops by linking IEG lessons to the project monitoring and evaluation. management portal. This will better support teams to When appropriate, engage a analyze the context of specific Bank engagements to be able broad range of stakeholders to adapt and adjust in a more flexible and timely manner. to help support the reform Recently, a mandatory task team leader (TTL) accreditation agenda and maintain the program has been launched using experiential and case- focus on key policy issues in based learning as well as on-the-job learning to strengthen the public domain. staff capacity in results-focused project design, Encourage emerging implementation, and client engagement. knowledge hubs to follow approaches along these lines. The use of programmatic Strengthening engagement WB: Agree WB: The Bank’s knowledge work (both RAS and non- approaches was important to and results through multi- RAS) are programmed in a strategic medium-term achieve outcomes, as shown year frameworks: Design framework defined by the CPS or CAS. CPSs/CASs are by examples of knowledge programmatic knowledge now designed to be results-based and include both lending services that enabled services where appropriate in and knowledge services. Furthermore, quality assurance technical assistance or a number of thematic areas. processes, results measurement, and monitoring have been economic sector work to Ensure that up to date strengthened for both RAS and non-RAS activities, follow up on initial work. By knowledge and/or high- including for just-in-time advice which is increasingly contrast, some projects quality research underpins demanded by our clients. It is therefore incorrect to imply launched in response to recommendations and that only non-RAS activities are strategic or that RAS demand from certain provide practical know-how activities are not strategic. agencies did not produce and customization of global strong results, sometimes practice. Develop and apply The recommendation to develop and apply principles to because these were one-off principles to ensure balance ensure balance between RAS and non-RAS activities has initiatives with poor between strategic (non- already been implemented. Recently adopted quality sustainability prospects. reimbursable) and assurance processes and systems have been used to xxix MANAGEMENT ACTION RECORD IEG Findings & IEG Recommendations Acceptance by Management Response Conclusions Management of Recommendatio n Staying engaged and reimbursable activities. strengthen the link between individual knowledge activities responsive in the and the CPS/CAS results. First, teams are now required to implementation phases of clearly outline the objectives and intended audience, along advisory services through with the intermediate outcomes that will be used to measure appropriate instruments can progress toward objectives. These objectives and outcomes help clients translate are required to contribute to one of the overarching recommendations from outcomes of CPSs/CASs, for those countries that have a sound analysis into actions CPS/CAS. For those countries that do not have a CPS/CAS that fit local political and (usually high-income countries), teams will still be required administrative constraints. to clearly indicate the objective and audience, intended By the same token, the focus contribution to development outcomes, and related on reimbursable advisory indicators of success. Second, these changes now apply also services to respond to client to RAS and new templates/tools have been designed for demand may entail a lack of teams to ensure that the quality and strategic orientation of continuous engagement in RASs are uniform across the Bank. Furthermore, RASs are some areas or lack of required to have the same level of quality and follow the coverage of thematic areas same processing as non-RAS ESW/TA. RAS related that may not rank sufficiently activities are also programmed within the CAS/CPS and high on the short-term guidelines have already been developed to define when a priorities of clients. The RAS should be undertaken (see the Op Memo-The Bank’s capacity to see the Provision of Reimbursable Advisory Services available on big picture and provide the OPCS website). These changes help to ensure the multi-sector development balance between RAS versus non-RAS in the Bank’s solutions—strength of the country programs. Third, at completion TTLs/task teams are Bank’s knowledge services required to complete a self-assessment of results achieved valued by clients—may thus with greater management oversight. Fourth, this assessment be eroded. is supplemented with a client feedback instrument xxx MANAGEMENT ACTION RECORD IEG Findings & IEG Recommendations Acceptance by Management Response Conclusions Management of Recommendatio n administered to both RAS and non-RAS activities. (In fact, the CFI was just completed for the FY12 cohort and the exercise will be launched in the coming months for the FY13 cohort of completed activities). Finally, teams are encouraged to capture/document in Bank systems the evidence of the contribution of their knowledge activity to results achieved. In addition to the above, a new programmatic instrument for ESW/TA is now on the Ops Portal. Programmatic approaches are encouraged in those situations where they are relevant and appropriate, i.e., in support of medium term reform efforts. Bank knowledge services Improving the results WB: Agree WB: Management has already begun to address the issues were not monitored and measurement system for raised by IEG. As noted above, knowledge services are now evaluated consistently in the knowledge services: The more closely linked to outcomes defined at the level of the sample of countries. In only Bank should ensure and CPS/CASs. The Bank has strengthened its results 17 percent of the knowledge monitor high-quality results framework for knowledge services and is now putting in services assessed was there frameworks that respond to place processing steps and guidelines to support monitoring at least a partial results client concerns, and link and quality. The self-assessment process has been framework in the CPS, more tightly knowledge strengthened to have more management oversight. A client allowing a tracking of the services tasks with country feedback instrument is now in operation. The next step is contribution of the activity to partnership strategies for the Bank to learn from the exercise and use it to inform the broader development milestones and outcome the automation of the process. This work is now ongoing. outcomes sought by the CPS. indicators. Use—and Going forward, the Bank will put in place an approach for Similarly, only 23 percent of continuously improve— ex-post assessment of knowledge work. In that regard, as the knowledge services implementation results plans for automation of the CFI proceed, we look forward to xxxi MANAGEMENT ACTION RECORD IEG Findings & IEG Recommendations Acceptance by Management Response Conclusions Management of Recommendatio n included, at least partly, monitoring systems that working with IEG on developing a framework for ex-post result indicators to track the track progress of knowledge evaluation which covers the full group of client facing achievement of the activity’s activities towards achieving knowledge products and internal knowledge products, i.e., outcome. Where monitoring outcomes in the results ESW/TA, IE, TE and KP. IEG has already initiated this and evaluation were better, framework of the country work with a focus on ESW/TA and we expect, as an initial knowledge services results partnership strategies. Ensure step, to have a framework along with advice on how to were more likely to be that all knowledge service strengthen the information content of the Ops Portal to achieved, most likely activities are timely, seek ensure adequate information for decision making, reflecting a link between independent validation monitoring and ex-post evaluation in the first quarter of M&E, knowledge services through systematic pointed FY14. quality, and impact. client feedback, and use such Implementation and results feedback to assess results monitoring systems are and improve results needed to track progress framework of knowledge toward mutually agreed services. Draw on outcomes and mitigate the appropriate evaluation risk of fragmentation and resources and develop tools loss of strategic focus to evaluate knowledge-based intrinsic in reimbursable programs. advisory services. Enhancing the Bank Group’s Staff incentives for WB: Partially WB: Various actions have been initiated to achieve greater success rate on knowledge excellence in knowledge Agree incentives for knowledge work. The increased focus on services will require that the services: Design appropriate results, client feedback and ex-post evaluation coupled with highest-caliber staff have incentives to promote staff the improved management oversight (from senior incentives to be in the engagement in knowledge management down to the task team) will help to place knowledge services activities. Move to a greater focus on results. Aside from the changes outlined “business”. There is a financial and budget above, we have instituted a routine management meeting xxxii MANAGEMENT ACTION RECORD IEG Findings & IEG Recommendations Acceptance by Management Response Conclusions Management of Recommendatio n perception at the Bank that framework that balances chaired by the managing director and attended by the lending experience is managerial incentives for regional operations directors/vice presidents and the essential for promotion, and lending and knowledge network vice presidents), to jointly review the knowledge the performance activities. Link funding to and lending portfolio. Starting from FY14, results from the management system does not country units to results from client feedback will be linked to or inform the MOU have enough incentives to country partnership strategy established between the managing director and the various promote knowledge monitoring framework rather vice presidents (Regions and networks). activities. Moreover, country than lending operations. unit budgets are organized The allocation of resources to support knowledge products around the supervision and is already driven by the country program's expected results. execution of lending Client and Bank teams determine what outcomes the Bank operations, with knowledge hopes to influence and then identify the activities needed to services being in support of achieve these outcomes. Resources are then allocated to such operations. Otherwise, support the required activities. In many cases these are stand knowledge services are more alone knowledge products that are unrelated to lending. like incidental items in the operation of country units. Management is committed to continue to strengthen and align incentives – including budget - to achieve excellence in knowledge services, and, in general, to achieve improved development results. However, it cannot commit at this stage to specific budget allocation principles, at the time when the entire budget process is being reviewed to align it to the new World Bank Group strategy under development. xxxiii Chairperson’s Summary: Committee on Development Effectiveness On June 24, 2013, the Committee on Development Effectiveness (CODE) met to discuss Knowledge-Based Country Programs: An Evaluation of World Bank Group Experience and the Draft Management Response. Summary The Committee welcomed both the Independent Evaluation Group’s (IEG) evaluation — which assessed knowledge-based activities in nine country programs selected from 48 knowledge-intensive programs supported by the World Bank Group — and management’s draft response. Members also appreciated the constructive engagement between IEG and management. The Committee agreed with the recommendations and findings, including that relevance to client concerns, technical quality of results, and reference to local policy context drive successful knowledge services; programmatic approaches and consistent engagement achieve better outcomes; stronger results monitoring measures real progress on agreed outcomes; and enhanced staff incentives to promote knowledge activities foster staff commitment. The Committee found the evaluation timely and useful in the context of the ongoing conversation on the Bank’s knowledge work, particularly on how knowledge is positioned within the institution’s overall strategy in the medium and longer term. Members noted that a core determinant of the Bank’s development effectiveness is the extent to which it can combine lending with knowledge and, hence, achieve results and find solutions. In this regard, members underscored that the Bank needs to ensure that knowledge services are more impactful on development outcomes at the country level — and hence, management must promote appropriate staff incentives in respect of knowledge work. Members noted IEG’s findings that results were very similar for free services and Reimbursable Advisory Services (RAS). Yet regardless of the type of knowledge service the Bank provides, members agreed that there needs to be clarity on the objective of a knowledge engagement in a client country, on the value it adds, and on the results framework that underpins its work. This would necessitate a clear understanding between the Bank and the client to ensure knowledge services are aligned appropriately. While it was recognized that RAS is a small part of the Bank’s knowledge services, members reinforced that the institution’s engagement in RAS should be in line with its overall strategy for a country. Members asked management to ensure the appropriate criteria are in place. Members appreciated that Bank management intends to continue to strengthen and align staff incentives to achieve excellence in knowledge services. Members acknowledged that management agreed in principle with the recommendation to move to a budget framework that better balances incentives for lending and knowledge activities, but accepted that management could not commit to specifics at this time since the budget process is under review to align it to the new World Bank Group strategy. Anna Brandt, Chairperson xxxv 1. Introduction Highlights  World Bank country programs have shifted toward more intensive delivery of knowledge services relative to lending in the FY05-11 period compared to FY98-04.  Differences in income levels and external financing requirements appear to be associated with the relative increase of knowledge services in country programs in the top half of a ranking of relative preponderance of knowledge services.  There is only a weak correlation between the relative Advisory Services intensity of International Finance Corporation (IFC) programs and the knowledge services content of World Bank programs because, while many countries tend to rely less on the Bank’s financing as they grow, opportunities for IFC investments may increase along with the size of the market and economic development.  Selected knowledge activities in the focus countries of the evaluation have been assessed against the criteria of relevance, technical quality, results achieved, and the sustainability of results. 1.1 Since the 1990s, the World Bank Group and its country clients have become increasingly cognizant of the importance of knowledge in development. In response to countries’ changing needs, the Bank Group has adapted its approach, governance, business practices, and strategic vision. It rebalanced its services away from a predominant focus on finance, toward more effective use of country experiences and global knowledge to meet client needs. The Bank repositioned itself in 1996, through the Knowledge Bank concept, with the aim of enhancing the technical quality of lending services, while developing new streams of knowledge services. This was followed in 1997 by the reorganization of the institution in a matrix management system—which was seen as a more effective means of leveraging the Bank’s global knowledge, its sector technical know-how, and its country-specific expertise gained from lending and policy advice. 1 The past decade saw a growth in the knowledge services the Bank provided. According to Knowledge for Development 2011, in FY10 the Bank allocated $606 million, or 31 percent of its administrative budget, for core knowledge services, up from 24 percent in FY02. 2 With its Advisory Services operations, the International Finance Corporation (IFC) fulfills a dual role too as a provider of finance to the private sector and knowledge conveyor. IFC’s expenditures on Advisory Services increased from $299 million in FY08 to $365 million in FY12. 3 1 CHAPTER 1 INTRODUCTION 1.2 In parallel, in the decade leading up to the 2008–09 global financial crisis, most of the Bank Group’s country clients faced an exceptionally favorable economic environment. Thus, several countries found that they needed less Bank financing, while others—especially new European Union (EU) members from Eastern and Central Europe—graduated from the World Bank’s financial assistance. However, most of these countries maintained a partnership with the Bank Group, based on economic and sector work (ESW) and on non-lending technical assistance (TA). Knowledge services have become a more important platform for the Bank to advise these countries on their development agendas. At the same time, arrangements for Reimbursable Advisory Services (RAS) have become more prominent in the Bank’s partnerships with country clients. 4 The growing volume of RAS signals that drawing on the Bank’s knowledge provides value. Relative Intensity of Knowledge Services in World Bank Country Programs 1.3 Multiple activities generate knowledge at the World Bank Group either as freestanding services or as knowledge embedded in lending operations. Although accurate estimates are difficult to come by, as noted in Knowledge for Development 2011, an estimated 10 percent of loan proceeds from projects financed by the Bank are being spent by borrowers on knowledge products that are similar to core knowledge services delivered through the Bank’s knowledge services. 5 Henceforth, knowledge products will refer to ESW and TA. 1.4 Depending on the intensity of use of knowledge services, Bank programs with country clients can be characterized as evolving along a continuum from programs dominated by finance and relatively lower intensity of knowledge services (lending-based programs, or LBPs) to programs with high knowledge content—where knowledge-based activities are at the core of the relationship (knowledge-based programs, or KBPs). However, drawing a line between KBPs and LBPs is arbitrary. First, the Bank typically offers a mix of lending and knowledge services to its country clients. This mix evolves over time, depending on country economic development and specific circumstances that may generate high or low external financing needs. Second, since knowledge is also generated by lending operations, it is difficult to assess whether a KBP generates more knowledge transfers to clients than an LBP with a country services budget of similar size without more detailed analysis. Subject to these caveats, a working definition of KBPs has been established for this evaluation with the use of indicators that help categorize World Bank country programs according to the predominance of knowledge services. Four indicators have been used (box 1.1): 2 CHAPTER 1 INTRODUCTION • The number of Bank lending operations in proportion to the number of knowledge products; • Resources allocated to Bank lending operations (cost of project preparation and supervision) in proportion to resources allocated to knowledge products; • Resources allocated to knowledge products as a percentage of country services budget; and • The number of knowledge products delivered. Box 1.1. Definitions, Use of Terms, and Selection of Knowledge Activities World Bank knowledge services and knowledge activities covered in this report consist of economic sector work (ESW) and technical assistance (TA). International Finance Corporation (IFC) activities are Advisory Services where the government was a client. Knowledge activities were selected in consultation with country management units, paying attention to links to strategic priorities in country partnership strategies. The selection of countries that included IFC Advisory Services was designed to illustrate complementarities and synergies with the Bank in those countries. The recommendations in this report are intended to apply to countries similar to those in the sample, and the small size of IFC Advisory Services reviewed does not provide a full illustration of IFC Advisory Service effectiveness. Note that given the sample of projects drawn on is relatively small in the case of IFC, the findings and recommendations should be taken to be most relevant to the World Bank’s knowledge work with some global applicability to the Bank group but not IFC specifically. Bank country programs were selected according to the preponderance of knowledge services using the following criteria: ratio of number of lending projects to number of knowledge activities, ratio of lending cost to knowledge service cost, share of knowledge services cost in country services budget, and number of knowledge activities. The combination of criteria intends to eliminate biases of any one criterion taken individually. Lending continues to play a crucial role in many middle-income countries, and this evaluation does not suggest that a linear transition from lending to knowledge services is the desired pathway for a given country. Two broad program categories with preponderance of knowledge services were distinguished: • Programs in countries where there is no pressing need for Bank financing and the partnership is thus designed with a strong focus on knowledge services. • Programs in countries where the Bank’s involvement revolves around non-lending activities because there is an insufficient foundation for a reform-oriented lending program. To assign countries with preponderance of knowledge services to one of the two categories, the evaluation used Country Policy and Institutional Assessment (CPIA) ratings over calendar years 2004-10. Countries with below- average CPIA ratings were put in the second category and removed from the sample. Focus was placed on the first half along a continuum of relative preponderance of knowledge services, with 48 countries fitting a working definition of knowledge-based programs. Additional criteria to narrow down the set of countries: • Primary criterion: upper-middle-income countries by per capita income level over 2005-10. • Secondary criteria to include countries: o mostly toward the high end of the knowledge services preponderance continuum o with diversified economic structure o with above-average size of knowledge services program o with no Bank lending services o with significant (average or above-average) use of Bank lending services o with fee-based services o that are Organisation for Economic Co-operation and Development members or new European Union members. • The selected focus countries are Bulgaria, Chile, China, Kazakhstan, Kuwait, Malaysia, the Russian Federation, South Africa, and Thailand. 3 CHAPTER 1 INTRODUCTION 1.5 Thus, abstracting from the knowledge services incorporated in lending operations, the lower the value of the first two indicators and the higher the value of the last two, the higher the knowledge content of country programs. Using these four indicators, a comparison of the periods FY05-11 and FY98-04 reveals a shift of Bank country programs toward more intensive delivery of knowledge services relative to lending (figure 1.1). Figure 1.1. Lending and Knowledge Activities in World Bank Country Programs (FY98-04 versus FY05- 11) 3.5 6,000 5,350 3.0 5,000 2.5 4,000 3,500 2.0 3,000 1.5 2,000 1.0 1,000 0.5 0.0 - FY98-04 FY05-11 # Lending/ # AAA AVERAGE Lending Cost/ AAA Cost AVERAGE AAA in % of Country Service budget TOTAL # of AAA Linear (TOTAL # of AAA ) Source: World Bank Business Warehouse data. 1.6 A ranking of 138 World Bank country programs based on the relative knowledge content (relative use of ESW and TA) over FY05–11 was established using the average of the country program scores on each of the four indicators (appendix A). A descriptive analysis of the 138 country programs categorized—in decreasing order—in terciles of the distribution of relative knowledge intensity shows that there are significant differences in the mix of Bank services among the three terciles (Table 1.1). Countries with programs categorized in the upper third of relative knowledge intensity had, on average, per capita GDP 2.5 times higher than countries in the other two terciles—among which there was no significant difference in per capita income. Countries in the upper third had, on average, a small current account surplus over 2005-10, compared to deficits of 6.7 and 5 percent of GDP in the middle and lower third of the distribution. Differences in income levels and external financing requirements are thus, probably, a driver of the higher mix of 4 CHAPTER 1 INTRODUCTION knowledge services relative to finance in the country programs in the upper third of the distribution. RAS for knowledge services were present in 16 of the 44 countries in the upper third, accounting on average for 50 percent of country services budgets for these countries (appendix A). On the other hand, reimbursement was present in only four countries in the middle third and three in the lower third. Table 1.1. Selected Corporate and Economic Indicators Associated with World Bank Country Programs Categorized According to the Relative Intensity of Knowledge Services (FY05–11) Percentile of the continuum of relative knowledge Corporate and economic indicators Upper-third Middle-third Lower-third percentile percentile percentile Average number of lending operations/number of 0.3 0.7 1.2 knowledge products Average resources for lending/knowledge services 0.8 2.0 3.5 resources Average knowledge services resources as 54.4 30.7 21.4 percentage of country service budget Number of knowledge products 2,144 2,224 982 Number of knowledge products as percentage of 40.1 41.6 18.4 total Bank knowledge products Average number of knowledge products by country 48.7 49.4 20.0 Total lending commitments (millions of US$) 70,728 130,011 47,945 Lending commitments as percentage of total Bank 28.4 52.3 19.3 lending commitments GDP per capita (in constant 2007 PPP US$, 12,683 4,840 4,703 average over calendar years 2005-10) Current account balance (as percentage of GDP, 0.8 -6.7 -5.1 average over calendar years 2005-10) Source: World Bank data and World Development Indicators. Note: PPP = purchasing power parity; GDP = gross domestic product. 1.7 In contrast, the patterns of IFC Advisory Services relative to investments are different from the patterns of Bank knowledge services intensity to lending. When IFC Advisory Services and investment operations are classified according to the continuum of knowledge services intensity of World Bank country programs, there is only a weak correlation between the Advisory Services intensity of IFC programs and the knowledge content of World Bank programs. In the upper third of knowledge intensive World Bank programs, IFC delivered, on average, 1.9 investment operations in proportion to Advisory Services. This proportion was only marginally higher in countries in the medium and lower thirds of the knowledge services intensity continuum of Bank programs (table 1.2). IFC’s mix of Advisory Services and investments was thus practically invariant along the relative 5 CHAPTER 1 INTRODUCTION knowledge services continuum of Bank country programs. One reason is that, while many of the World Bank’s country clients tend to rely less on the Bank’s financing and more on its knowledge services as they grow, opportunities for IFC investments may increase along with the size of the market and economic development. At the same time, with the improvement of the quality of regulations in countries with robust private sectors, the scope for IFC Advisory Services may decrease. To further complicate matters, the funding of IFC Advisory Services by donor trust funds may shape the allocation of services toward specific countries or regions, not closely related to patterns of economic development. Table 1.2. IFC Advisory Services and Investment Operations Associated with World Bank Country Programs According to Their Relative Knowledge Service Content (FY05-11) Percentile on the continuum of relative knowledge service intensity of World Bank country programs Indicators Upper-third Middle-third Lower-third percentile percentile percentile Average number of IFC Advisory Services 13.0 16.6 5.7 Average number of IFC investment operations 17.0 24.4 9.6 Investment operations/Advisory Services 1.9 2.0 2.1 Source: World Bank Business Warehouse data. Objectives, Scope, and Methodology of the Evaluation 1.8 As a growing number of emerging market economies rely less on World Bank financing, opportunities for Bank knowledge transfers to clients through lending have declined. Despite the spike in lending during the 2008–09 global financial crisis, opportunities for knowledge transfers through lending are likely to diminish further in the future, especially in upper-middle-income countries with no pressing financing needs or with improving access to diversified sources of finance. The Bank’s ability to contribute to development policy and poverty reduction through finance may thus become more limited. Knowledge-based country programs in upper-middle-income countries will therefore become a more important platform for the Bank to advise clients on their development agendas. 1.9 The objective of this evaluation is to learn lessons from diverse practices in the World Bank’s knowledge-based programs with high (Kuwait) and upper- middle-income countries, to help leverage the Bank’s global knowledge most effectively to meet the needs of these countries that are not in pressing need of finance and mainly rely on knowledge services. 6 CHAPTER 1 INTRODUCTION 1.10 Some key questions the evaluation will answer in pursuing this objective are the following: • How best can the Bank Group maintain its relevance in knowledge-based partnerships? • How does the World Bank position itself as a development partner of choice in upper-middle-income countries? • What are the success factors for knowledge services to achieve expected results? • How can the World Bank add more value for clients through appropriate scope and instruments of knowledge-based country programs? 1.11 Over the past few years, the Independent Evaluation Group (IEG) has conducted three evaluations that address some aspects of knowledge services provided by the Bank Group: Development Results in Middle-Income Countries: An Evaluation of the World Bank’s Support (in 2007); Using Knowledge to Improve Development Effectiveness: An Evaluation of World Bank Economic and Sector Work and Technical Assistance, 2000–2006 (in 2008), and Independent Evaluation of IFC’s Development Results: Knowledge for Private Sector Development (in 2009)—see Box 1.2. This evaluation fills a gap in previous work because it assesses the performance of knowledge services in the context of country programs that revolve primarily around such services. 1.12 As the primary objective of the evaluation is to understand how the World Bank can most effectively work with country clients that do not rely on Bank financing, and as IFC’s engagement through Advisory Services follows a different pattern from the World Bank’s KBPs, lessons learned are not applicable to IFC. However, the evaluation aims to draw lessons on how best the synergy between the Bank’s knowledge services and IFC Advisory Services could be mutually reinforced in promoting private sector development in upper-middle-income countries with Bank KBPs. These lessons are not intended as a comprehensive assessment of IFC- World Bank synergy but rather of the way that synergy plays out in countries with knowledge-based World Bank programs. 1.13 To deliver on its objective, the evaluation uses focus countries selected from among the countries that make relatively more intensive use of the Bank’s core knowledge services. The selected countries are Bulgaria, Chile, China, Kazakhstan, Kuwait, Malaysia, the Russian Federation, South Africa, and Thailand. The evaluation has also taken into account IFC Advisory Services in China, Kazakhstan, Russian Federation, and South Africa. The nine selected countries are high (Kuwait) and upper-middle-income countries with a high share of knowledge services in their 7 CHAPTER 1 INTRODUCTION programs. They were selected to include countries with a diversified economic structure, countries with an above average size of knowledge services program, countries with no Bank lending, countries with fee-based knowledge services, and countries which are Organisation for Economic Cooperation and Development (OECD) or new EU members. The selection was designed to provide useful illustrations of knowledge service effectiveness in the selected countries, not to provide a sample for statistical projection to overall knowledge service assistance to those countries or the full set of Bank clients. Furthermore, the selection of countries that included IFC Advisory Services was designed to illustrate complementarities and synergies with the Bank in those countries, not to provide a full illustration of IFC Advisory Services effectiveness in those countries. Box 1.2. Previous IEG Evaluations of Knowledge Services The main recommendations of the three evaluations of knowledge services conducted by the Independent Evaluation Group (IEG) are as follows: Development Results in Middle-Income Countries: An Evaluation of the World Bank’s Support  Draw on country capacity and develop each country’s own expertise  Demonstrate best practice in its support to countries  Increase the Bank’s agility in support of countries  Make the most of cooperation across the Bank, International Finance Corporation (IFC), and the Multilateral Investment Guarantee Agency (MIGA) to successfully offer clients a more effective package from its combined resources. Using Knowledge to Improve Development Effectiveness: An Evaluation of World Bank Economic and Sector Work and Technical Assistance, 2000–2006  Reinvigorate the Bank’s mandate to maintain a strong knowledge base  Ensure that ESW tasks in countries funded by the International Development Association (IDA) are adequately resourced, even if it means fewer economic and sector work (ESW) tasks in some countries  Enhance the institutional arrangements for undertaking ESW and technical assistance by ensuring substantive task team presence in country offices, particularly in countries with low institutional capacity, and by formulating a dissemination strategy at the concept paper stage  Build on client preferences, including clients’ feedback after completing the task  Take the results tracking framework seriously. Independent Evaluation of IFC’s Development Results: Knowledge for Private Sector Development  Effectively manage the tension between protecting the portfolio and responding to opportunities during the crisis  Set out a clear vision and business framework for Advisory Services that is more closely linked with IFC’s global corporate strategy  Pursue more programmatic Advisory Services interventions  Over the long term, pricing of Advisory Services should reflect the value and impact (that is, not just the cost)  Strengthen Advisory Services performance measurement and internal knowledge management. 8 Table 1.3. Distribution of Bank Knowledge Services and IFC Advisory Services Across Countries and Thematic Sectors (FY05–11) Poverty Sample Economic reduction/ All know Sample All IFC IFC Agricul Edu- Private Infra- policy/public social ledge knowledge Advisory Advisory -ture cation/h sector structure sector protection services services Services Services Country (%) ealth (%) (%) (%) governance (%) (%) (number) (number) (number) (number) Bulgaria 5.1 5.1 25.6 28.2 25.4 10.3 39 18 5 0 Chile 4.1 26.5 28.6 10.2 22.4 8.2 49 25 2 0 China 9.8 6.3 18.5 39.0 17.6 8.8 205 50 70 13 Kazakhstan 3.1 7.0 29.7 11.7 37.5 10.9 128 49 5 4 Kuwait - 9.8 51.2 14.6 24.4 - 41 15 0 0 Malaysia 3.2 16.1 16.1 9.7 54.8 - 31 13 0 0 Russian 8.6 14.8 18.8 24.2 18.8 14.8 128 49 71 9 Federation South Africa 12.5 5.4 26.8 28.6 23.2 3.6 56 21 26 8 Thailand - 16.2 16.2 18.9 32.4 16.2 74 26 6 0 All AAA (%) 6.3 10.7 23.6 24.1 25.7 9.7 - - - - All (number) 47 80 177 181 193 73 751 - 185 - Sample (%) 7.5 11.7 32.7 9.0 26.3 12.8 - - - - Sample 20 31 87 24 70 34 - 266 - 34 (number) Note: AAA = analytical and advisory activities. 1.14 The World Bank delivered 751 knowledge products to the focus countries over FY05-11 while IFC delivered 185 Advisory Service projects to both private and government entities over the same period—see Table 1.3 for a breakdown of Bank knowledge services by thematic areas. 6 Approximately three-quarters of all knowledge services were delivered in the areas of private sector, infrastructure and economic policy, and public sector governance. There is wide variation among the countries in the distribution of knowledge services across sectors. For example, while Infrastructure represents 39 percent of knowledge services in China, it represents approximately 10 percent in Malaysia. And whereas private sector constitutes 51 percent of the knowledge services in Kuwait, it represents 16 percent in Malaysia and Thailand. The variation observed in the distribution of knowledge services reflects different country priorities. 1.15 In view of the large number of knowledge products delivered by Bank programs in the focus countries, the evaluation uses a purposively selected sample of knowledge products delivered over FY05–11. The criteria used to select the knowledge products and IFC Advisory Services are described in Appendix B. The sample includes 266 World Bank knowledge products and 34 IFC Advisory Service projects (see table 1.3). In order to capture activity synergies and possible complementary results, selected knowledge products and IFC Advisory Services were grouped, whenever possible, into “knowledge activities” with similar objectives or thematic focus. Thus, the sample of Bank knowledge services was 9 CHAPTER 1 INTRODUCTION reduced from 266 to 196 activities and the sample of IFC Advisory Services was reduced from 34 to 32 activities. Therefore, the number of “knowledge activities” reviewed by this evaluation is 228. The list of the “knowledge activities” reviewed and the underlying knowledge products and IFC Advisory Services is in Appendix B. As with the selection of countries, the selection of knowledge was intended to illustrate knowledge service effectiveness and synergies, not to project results statistically to the full range of Bank knowledge services and Bank/IFC Advisory Services synergies of knowledge services in the selected countries. The selected knowledge activities in the nine focus countries have been assessed against four criteria: • Relevance is intended to assess the extent to which the knowledge activities were suited to the priority needs of the recipients and to the achievement of key development goals of the country client • Technical quality of the knowledge activities and their effectiveness in leveraging the Bank’s global knowledge and conveying relevant and customized expertise to recipients • Results are intended to assess whether the knowledge activities attained their objectives, whether results can be traced and attributed to the Bank, and whether the process through which the knowledge services were delivered and shared with various stakeholders supported the achievement of results • Sustainability of results as a measure of whether the knowledge activities are likely to have lasting impacts (on policies, capacity, or institutions) after their completion, and whether an appropriate framework is in place to monitor results over time. 1.16 This review is not meant to be a full-fledged evaluation of country program performance. As in most focus countries the review was conducted while the most recently approved knowledge-based country programs was under implementation, prior to the completion of their cycle, and covered only a subset of the knowledge services completed under the country programs. 1.17 IEG has conducted evaluation of knowledge services in various areas, beyond the Bank’s lending operations, building on the knowledge service assessment methodology used by the Bank’s Quality Assurance Group (QAG). The methodology applicable to knowledge services has been tested by IEG in thematic evaluations. 7 This evaluation draws on IEG’s assessment methodology for knowledge services, with appropriate modifications. The evaluation template for the selected knowledge activities was designed to reflect the four assessment criteria (Appendix F). The necessary information was collected by the evaluation team through desk reviews of country partnership strategies and project documents and 10 CHAPTER 1 INTRODUCTION through interviews with Bank staff, government officials, and other stakeholders, including representatives from civil society organizations. 1.18 The report is organized in five chapters including the present chapter. Chapter 2 reviews the relevance and strategic positioning of the World Bank in knowledge-based partnerships. Chapter 3 reviews the technical quality of the knowledge services delivered. Chapter 4 evaluates the results of the knowledge activities and their sustainability, as well as the synergy between World Bank knowledge services and IFC Advisory Services in promoting private and financial sector development. Chapter 5 draws conclusions and discusses the lessons learned from the evaluation. 11 2. Relevance of the World Bank in Knowledge- Based Partnerships Highlights  In most of the focus countries there was a structured partnership for the delivery of the World Bank’s knowledge-based program, although in some cases there was no results framework.  In a majority of World Bank knowledge services and IFC Advisory Services reviewed, the issues addressed were relevant to the client, while the design of the activities was appropriate to meet their objectives.  In more than a quarter of knowledge services and Advisory Services, however, there were issues pertinent to the client that the activities were at least partly unable to address—when reports provided very broad suggestions, with lack of customized solutions aligned with the clients’ demands.  The knowledge services delivered by the Bank Group have had multiple uses, with raising stakeholder awareness being predominant. Client participation was generally high—an indication of client interest and relevance of the work conducted.  Full dissemination of Bank knowledge services was conducted in slightly more than half of knowledge services reviewed. The confidentiality often imposed on Bank knowledge services cofinanced by clients or through Reimbursable Advisory Services (RAS) has lowered the rate of dissemination of Bank knowledge services.  Key strengths of the Bank’s knowledge services acknowledged by counterparts include the ability of benchmarking against good practice, reputation and credibility, ability to customize to the local context, and capacity to deliver multisector development solutions.  The Bank’s recognized capacity to customize and deliver multisector solutions, however, might be challenged when there is insufficient knowledge products to gain appropriate depth and breadth of country knowledge.  There are some good examples where the Bank’s knowledge services facilitated South-South exchanges, but more can be done as the Bank’s geographic, thematic, and organizational fragmentation prevents realizing the full potential of such exchanges.  The Bank’s positioning as a strategic partner and policy advisor with respect to other knowledge providers is generally favorable. 2.1 This chapter assesses whether the Bank Group has remained relevant in countries with knowledge-based World Bank country programs and how it is 13 CHAPTER 2 RELEVANCE OF THE WORLD BANK IN KNOWLEDGE-BASED PARTNERSHIPS positioning itself in countries that make relatively little use of its financial services. The chapter first assesses the overall fit of knowledge-based country programs with the development challenges of these countries and looks at the relevance of the design of the knowledge services delivered. Next, it looks at the relevance of the knowledge services delivered from the perspective of their use and the extent of client participation in the activities. Finally, it assesses how clients perceive the strengths and weaknesses of knowledge services delivered by the Bank Group in the focus countries and what it means for the strategic positioning of the Bank Group with respect to other providers of knowledge and advice. Fit of World Bank Knowledge-Based Country Programs with Country Challenges 2.2 Despite diminishing financing needs of several focus countries, owing to strong growth and the accumulation of substantial financial buffers (Chile, China, Kazakhstan, Kuwait, Malaysia, and Russian Federation) or access to other sources of financing, such as European Union (EU) structural funds (Bulgaria), overall, the Bank has maintained strong engagement through a change in country partnerships toward knowledge services. The World Bank faces multiple client demands for knowledge services in these countries and, to remain relevant, responds with different products, delivery practices, and under various financing arrangements. In several cases, International Finance Corporation (IFC) Advisory Services respond to similar client demands and often complement the Bank’s knowledge activities. Based on the Independent Evaluation Group’s (IEG’s) review of World Bank knowledge services in the focus countries, client demand generally falls into four categories: • Customized development solutions • Experience sharing and innovative ideas • Capacity development • Public knowledge goods. 2.3 Customized development solutions. These are studies or technical assistance (TA)—delivered mostly as Reimbursable Advisory Services (RAS)—that fill knowledge gaps in areas where counterparts need to develop a strategy or to take action. Relevance, timeliness, and customized advice through actionable recommendations are seen by counterparts as key. Such was the case in Chile, with TA on the management of financial resources of the pension fund or with studies on quality assurance in education and the program of student loans. In Bulgaria, the Bank’s new partnership provides customized policy support for the implementation of the Europe 2020 strategy and for accelerated absorption of EU funds. In the 14 CHAPTER 2 RELEVANCE OF THE WORLD BANK IN KNOWLEDGE-BASED PARTNERSHIPS Russian Federation, there is a growing demand for the Bank’s global knowledge for development solutions at the subnational level. At the same time, IFC provided advice on how to improve business access to land by using knowledge generated from regional surveys. In China, where knowledge services are being delivered along with a sizable lending program, counterparts sought the Bank’s advice for solutions to specific problems—such as a rural health system and energy-efficient transport system, based not only on rigorous analysis but also on demonstrations of their applicability to China through pilot lending projects. In parallel, IFC helped draft the personal bankruptcy law, design the credit reporting system, and develop a modern secured transaction systems. While based on experience in other countries, these initiatives had to be designed to take into account the Chinese context, including the state of current institutions and attitudes. 2.4 Experience sharing and innovative ideas. The Bank faces strong demand for studies or seminars that explore issues in an area where counterparts have not taken a position and where the Bank functions as a “sounding board” or connects counterparts to cutting-edge international expertise through its broadly recognized convening power. IFC also faces similar demands for Advisory Services intended to share IFC experience and global knowledge or demonstrate the feasibility of a solution to a policy issue. In Kazakhstan, the Joint Economic Research Program (JERP) is anchored in policy analysis, good practice options notes, and brainstorming sessions with high-level officials on a variety of topics where the government needs to form a view. The brainstorming sessions have become a critical platform to share opinion and help the authorities systematically think through issues with substantial analytical support from the Bank. In Bulgaria, the Bank’s knowledge services often served to validate the government’s views and have been used, in particular by the Ministry of Finance, to strengthen the dialogue with line ministries. This was the case in Chile as well, with Bank reports on policy assessment lessons from Organisation for Economic Co-operation and Development (OECD) countries or on decentralization. In some cases, there is a high sensitivity to embracing Bank policy recommendations due to political economy considerations, as, for example, in South Africa, especially on labor issues. 2.5 In Malaysia, the requests reflected a desire on the part of the client for the Bank to provide the intellectual underpinning, through theoretical and empirical evidence, for policy measures that had been debated internally. Studies addressed key aspects of building a high-income economy—raising productivity, increasing the added value of industry, improving equality, improving corporate governance, and raising standards. China also looked to the Bank for new concepts, advanced methodologies, and modern practices in formulating its own long-term strategies. Such advice tended to be provided either in the form of high-profile reports, such as 15 CHAPTER 2 RELEVANCE OF THE WORLD BANK IN KNOWLEDGE-BASED PARTNERSHIPS China 2030 and the Guangdong Reducing Inequality knowledge product, or as informal policy notes destined for a small circle of policy makers. In parallel, with the government’s focus on regional development, IFC helped introduce in Sichuan province several best practices and techniques in the area of investment promotion, such as use of cluster and value chain analysis. Successful demonstration of these practices in Sichuan would provide other regions with investment promotion models to consider. In the Russian Federation, the Bank delivered knowledge services on economic diversification, competition, and innovation at a time when these topics were not on the agenda of policy makers. At the same time, the World Bank and IFC jointly developed the subnational Doing Business Survey and monitoring system targeted toward the specific needs of local governments. The authorities intend to use the indicators for evaluating regional performance. 2.6 Capacity development. Very often the client is interested in knowledge that helps build capacity through training, networking, or access to international best practices. This service is typically delivered in the form of RAS, but not always. Such demand was strong in South Africa, with RAS for land reform but also TA for macroeconomic modeling. There are multiple examples of TA for capacity development in the focus countries: for risk-based pension and insurance supervision in Chile; for risk-based tax audits, the analysis of tax incidence and for anticorruption in Thailand; for performance-based budgeting in the Russian Federation and Bulgaria; and on regional innovation systems and best practices in nurturing innovation in enterprises in the Russian Federation. IFC also responded through Advisory Services to similar capacity development demands: to investment promotion agencies in two earthquake-affected provinces in China on how to restore the flow of private investments and speed economic recovery; to several private sector–run entrepreneurship programs in South Africa, including one that focused on women entrepreneurs. IFC’s South Africa work on pilot credit bureaus, focusing on microfinance clients, is being replicated by the private sector, with the Bankers’ Association planning to embark on similar schemes. IFC used experience from other countries to design a project that addressed risks emanating from negative attitudes toward information sharing. 2.7 Public knowledge goods. In addition to these services overwhelmingly driven by client demand, in all focus countries the Bank provided knowledge services as public goods (see box 2.1), typically fully funded by the Bank. These include Financial Sector Assessment Program (FSAP) reports in several countries and Reports on the Observance of Standards and Codes (ROSCs) with frequent focus on corporate governance and accounting and auditing. Investment Climate Assessments (ICAs) have been very much in demand in almost all focus countries of the evaluation. Regularly published Economic Monitoring Reports are another 16 CHAPTER 2 RELEVANCE OF THE WORLD BANK IN KNOWLEDGE-BASED PARTNERSHIPS knowledge product that functions as a public good. In Thailand, the Bank has produced the Economic Monitor twice a year for over a decade. The report is broadly disseminated and the Bank conducts regular consultations with the key stakeholders prior to publication. In Malaysia, too, the bank produces the Economic Monitor with a macroeconomic update and an analysis of a topical issue. The Bank selects the theme for each publication in consultation with the government, academics, and local think tanks. Box 2.1. Public Knowledge Goods Bank knowledge services primarily driven by client demand typically respond to needs at different stages of the policy-making process in a given area the client is considering, from looking into global experience (“experience sharing and innovative ideas”), through designing an approach or strategy (“customized development solutions”), and to implementing strategies (“capacity development”). In addition, the Bank provides “public knowledge” services not driven by those immediate client needs and usually fully funded by the Bank. For example, Financial Sector Programs, Reports on the Observance of Standards and Codes, or Investment Climate Assessments may not respond to immediate drives to reform the financial sector, corporate governance, or the investment climate, and yet provide broader “public knowledge” that may eventually lead to policies in those areas. Similarly, Economic Monitoring Reports or development policy reviews may cover a range of issues of public interest that extend beyond the focus of current public policy making and serve as tools for broad country-level consultation. These may be relatively more relevant from the broad perspective of country development challenges than from the perspective of the country’s more immediate policy focus. The Malaysia Economic Monitor provides an example of such “public knowledge good,” where the Bank covers macroeconomic developments and addresses a topical issue following consultation with various stakeholders. The April 2012 issue, for example, covered macroeconomic performance and policies; growth prospects and progress of structural reforms; and a special section on labor markets and policies, including social insurance. Accordingly, the report provides information of broad interest to the public that may eventually feed into policy-making efforts. These knowledge products help the Bank maintain the broad development focus that accounts for much of its appeal to clients. 2.8 The Bank has been working in the focus countries with flexible financing arrangements and delivery mechanisms for knowledge services. Clients often participate in the financing of knowledge work, which indicates that the Bank’s knowledge service products are relevant to the clients. In Kuwait, the Bank’s knowledge services program is entirely delivered through RAS, as is mostly the case in Malaysia. In both countries, the Bank does not deliver lending services, but there is selective lending in Bulgaria, Chile, Kazakhstan, Russian Federation, South Africa, and Thailand. In Chile and Kazakhstan, knowledge activities are being designed and delivered through jointly financed programs. In the Russian Federation, the 17 CHAPTER 2 RELEVANCE OF THE WORLD BANK IN KNOWLEDGE-BASED PARTNERSHIPS knowledge services program over FY08-11 was funded up to 46 percent through RAS, with a strong focus on knowledge services for regional governments and other nonfederal institutions. In Bulgaria, China, South Africa, and Thailand, knowledge services have been mainly financed through the Bank’s administrative budget. In China, in particular, the Bank’s country program remained firmly tied to lending, with knowledge services supporting and being supported by lending services. In Bulgaria, a sizeable RAS program is about to be initiated with focus on facilitating Bulgaria’s convergence with EU standards through the EU 2020 Strategy. 2.9 World Bank knowledge-based country programs typically face a tension between the need for flexibility to respond to rapidly changing client needs and the need for a structured partnership with relevance to the country’s development challenges, able to track development results. In eight of the evaluation’s nine focus countries there was a structured partnership for the delivery of the World Bank’s knowledge-based program. Country partnership strategies (CPSs) in these countries contained strategic pillars aligned with the governments’ programs, thus ensuring their overall strategic relevance. The pillars were comprehensive enough to encompass several areas of Bank engagement and the CPS documents highlighted the knowledge and lending services to be provided in each area. In Malaysia, the Bank has not had a CPS since 1999. The Bank entered into a framework agreement with the government in 2009, which defines the legal responsibilities of both parties for largely RAS but does not identify strategic pillars or outcomes. Other countries with knowledge-based World Bank programs not covered by this evaluation, such as Saudi Arabia, also have no formal CPS. 2.10 In Thailand, the CPS had not been updated for a long time, with the FY03-05 Country Assistance Strategy (CAS) carried forward through FY10. 8 The centerpiece of the relationship was the Country Development Partnerships (CDPs), revolving around TA and ESW. There was at least one CDP for each of the CAS pillars. The CDPs formed a practical platform for collaboration that allowed Bank interventions to be adjusted to the needs of the moment. In response to the need for flexibility in addressing evolving client demands, the 2004 Kazakhstan CPS was open-ended by design. It was expected to remain in force as long as it remained relevant to the client, provided value added to the country’s development process, and addressed the challenges of the policy environment, with a regular progress report assessing CPS relevance against these criteria. 2.11 Most CPS documents usually included a results framework with outcome indicators specific to each area of engagement. Results indicators could be elaborate or generic, in the latter case making it difficult to attribute results to the Bank’s 18 CHAPTER 2 RELEVANCE OF THE WORLD BANK IN KNOWLEDGE-BASED PARTNERSHIPS activities (for example, GDP growth above 5 percent in Chile; preventing real appreciation of the ruble in excess of 7 percent, or keeping federal expenditures below 18 percent of GDP in the Russian Federation). In Thailand, a rather complicated results framework, with measurement, partnership, implementation, and risk elements, was presented in the FY03-05 CAS and intended to be used for each CDP. The framework agreement with Malaysia does not provide a results framework for assessing the impact of the Bank’s knowledge services. 2.12 More recent CPSs (as, for example, the FY07-11 Russian Federation CPS, the FY11-13 Bulgaria CPS) have a detailed results framework that links planned knowledge services to specific outcomes expected to be influenced by the Bank’s program and provide milestones to track implementation progress. Generally, few measurable indicators to trace progress toward the milestones, and CPS outcomes are being provided. In Kazakhstan, the FY12-17 CPS presents a results framework, including major milestones and outputs and expected outcomes for each priority area in the government’s strategy supported by the Bank program. The results framework lists proposed activities in support of each of these outcomes. The new CPS envisions a programmatic approach aimed at strengthening the strategic focus of the JERP. This would be achieved through improving task sequencing, emphasizing interconnected tasks to better address policy linkages, and better tracking of impact. 2.13 The CPS is also an important instrument in guiding the selection of IFC Advisory Service projects in a country. In China, projects in access to finance and regional development not only aligned with the CPS but also made a significant contribution to the achievement of expected outcomes. In the Russian Federation, the IFC work on land transactions and markets was part of a program, along with a World Bank investment loan, to achieve one of the stated outcomes in the CPS. In South Africa, IFC focused its projects on micro, small, and medium enterprises (MSME) in line with the CPS emphasis on MSME. In general, the quality of the CPS documents and their results frameworks provide strategic direction to Advisory Services work and enable a more programmatic and longer-term approach to achieve higher development impact than would be the case when project selection is made on an ad hoc and opportunistic basis. 2.14 IFC Advisory Service projects would have to be justified based on several criteria articulated in the concept and approval documents. 9 There is no formal coordination mechanism that would enable Bank staff or management to provide inputs to the decision making. In practice, IFC country representatives typically coordinate with World Bank country managers or financial and private sector development staff in the field. A formal coordination mechanism between IFC 19 CHAPTER 2 RELEVANCE OF THE WORLD BANK IN KNOWLEDGE-BASED PARTNERSHIPS Advisory Services and Bank knowledge services would help improve alignment of Bank Group knowledge activities with CPS objectives. For example, the work in South Africa on competition policy and administrative barriers would have benefited from such a mechanism. 2.15 In most of the focus countries, especially when the Bank delivered RAS, the content of the program was decided in consultation with the authorities—an indication of its overall relevance to the client. In Chile, the Bank consults with line ministries and other agencies to identify sector priorities but the selection of activities under the Joint Studies Program (JSP) is decided jointly with the Director of Budgets of the Ministry of Finance (DIPRES), which has a strong influence on the content of the JSP. The Bank commits few of its own resources to knowledge services outside of the JSP. Cofinancing of the JSP by the government indicates that the authorities find value in the Bank’s knowledge services, which ensures that the knowledge work conducted through the JSP remains relevant as it is demand- driven. In Kazakhstan too, the knowledge-based country program is anchored by the JERP. That program has contributed to increasing the relevance of the Bank work in Kazakhstan. 10 In Malaysia, the requests for reimbursable knowledge services are coordinated by the Economic Planning Unit of the Prime Minister’s Department. Seventeen agreements have been concluded since 2009. 2.16 Coordination of the knowledge-based program was weaker in Thailand and in South Africa, as knowledge services were provided to various counterparts on demand, sometimes piecemeal, funded by the Bank’s budget. However, knowledge activities were aligned with the priorities of the CPSs. Similarly, in the Russian Federation, coordination was comparatively weaker than in other countries as the vast majority of RAS—covering 68 percent of the RAS budget over FY08-11—was delivered to regional governments and other nonfederal institutions. 2.17 The relevance of the Bank Group’s knowledge services appears to be broadly confirmed by IEG’s review of Bank knowledge activities and IFC Advisory Services in the focus countries. The issues addressed in the majority of Bank knowledge activities and IFC Advisory Services reviewed were identified as a development constraint or opportunity, either in the CPS or in previous sector work, lending, or policy dialogue (Table 2.1). The activities reviewed were demand-driven as the client had requested or commissioned the vast majority of Bank knowledge services and most of the IFC Advisory Services. IEG found the design of the majority of reviewed Bank knowledge services and IFC Advisory Services appropriate to meet their objectives, which corroborates their relevance. At the same time, some knowledge services could have been better tailored to client needs. IEG found that in more than a quarter of knowledge services and Advisory Services there were issues 20 CHAPTER 2 RELEVANCE OF THE WORLD BANK IN KNOWLEDGE-BASED PARTNERSHIPS relevant to the client that these knowledge activities were at least partly unable to address (Table 2.1). Table 2.1. Relevance Indicators of Knowledge and Advisory Services Reviewed by IEG (Percent of Reviewed Activities) Indicators Categories Percentage NA No Partly Yes World Bank 6.1 12.2 9.7 71.4 Issues addressed by knowledge activity identified as Knowledge Services development constraint or opportunity IFC Advisory Services 0 3.1 6.3 90.6 World Bank 1 9.7 8.2 81.1 Activity requested by the client Knowledge Services IFC Advisory Services 0 21.9 18.8 59.4 World Bank 1 6.6 92.3 Design appropriate to meet objectives Knowledge Services IFC Advisory Services 6.3 12.5 81.3 World Bank 6.1 67.3 14.8 11.2 Were there any relevant issues to the client the Knowledge Services knowledge activity was not able to address? IFC Advisory Services 3.1 68.8 12.5 15.6 Source: Independent Evaluation Group (IEG) knowledge activity reviews. Note: NA = Not applicable. Some questions on the tasks (such as, Was the activity requested by the client?) were more objective than others (such as, Was the design appropriate to meet objectives?) and accordingly were less subject to observation errors. The indicators in the table average the responses across all tasks and are likely to average out the observation errors of responses on the individual knowledge activity. 2.18 Most often, Bank knowledge services were not able to address issues relevant to the client because reports provided very broad suggestions, with little operational or directional value (see also chapter 3, Technical Quality of Knowledge Services). This was the case for a report on private sector development in China produced in 2012 with focus on innovation systems. As the initial draft was very broad and did not meet client expectations, the final report was delivered with a one-year delay, after the publication of the 12th five-year plan, and could not inform it, as had been intended, in any meaningful way. In 2010, a broad comparative analysis of SME finance across Eastern European countries was conducted in the Russian Federation. Despite identifying major obstacles to SME financing, however, the report was short on specific measures that the authorities could consider implementing. In Kuwait, the Bank produced a report to support the National Strategy for Economic and Social Development, but the design did not meet client expectations of an actionable plan to implement the “2010-35 vision” strategy previously elaborated by a global consulting firm. Instead, the report was seen as theoretical, with vague recommendations under each pillar and no specific “how to” guidance. 21 CHAPTER 2 RELEVANCE OF THE WORLD BANK IN KNOWLEDGE-BASED PARTNERSHIPS 2.19 In some cases relevance was hampered by the lack of customized solutions aligned with client demand. For example, in Chile, the authorities were interested in strengthening the financial asset management system within the existing institutional framework, without a dramatic institutional change in the short term, but they were prepared to consider a substantial institutional change in the long term. The Bank’s 2010 knowledge activity on financial asset management insisted from the beginning on drastic institutional change—creating a new independent financial management unit outside the Ministry of Finance—and never came to grips with providing a short-term operational solution. Sometimes the Bank’s knowledge services, while relevant to the client overall, missed specific relevant issues that would have further increased its value. For example, in Bulgaria, the 2012 report on public expenditure for growth and competitiveness would have been of greater value to the client had it included an analysis of “nonprice competitiveness” to provide input for the authorities’ discussions with the EU. In Thailand, the Bank’s TA on tax incidence addressed relevant issues for counterparts but did not address tax incidence by industry, which the Tax Revenue Department would have wished to also assess. Overall, additional Bank effort should go into ensuring that clients’ expectations from knowledge services are clearly understood so that the end products meet these expectations or appropriate follow-up work is done. In China, the IFC Advisory Services on leasing changed objectives over the life of the project; a programmatic approach would have been more appropriate, with front-end analytical work. The Sichuan Recovery Investment Program did not have flexibility in objectives and design to respond to rapid changes in priority or need given the emergency nature of the project. In South Africa, the design of support to the Women’s Entrepreneurship Program did not consider financial sustainability and the program could not continue after IFC support stopped. Scalability was thus limited as the program covered fewer than 200 women and was not replicated. 2.20 In many cases, IFC Advisory Service projects are developed to demonstrate to clients the importance of issues or the feasibility of particular solutions, and they succeeded in doing so. In China, for example, the initial project on secured transactions was meant to raise awareness of the benefits of establishing a regime, leading to eventual government adoption of a modern system. In the Russian Federation, the initial work on reducing administrative barriers—a study and a dissemination strategy—was meant to get government buy-in to reforms, which were supported by IFC Advisory Services. However, work on competition in South Africa, which was not well received by counterparts, illustrates the risk of embarking on projects with weak client ownership. 11 22 CHAPTER 2 RELEVANCE OF THE WORLD BANK IN KNOWLEDGE-BASED PARTNERSHIPS Use of World Bank Group Knowledge Services and Stakeholder Engagement 2.21 As the Bank Group responded to multiple demands in the focus countries, the knowledge services delivered have generally had multiple uses. Raising stakeholder awareness was the predominant use of Bank knowledge activities (figure 2.1). Use for other purposes—such as reforming government programs, helping reach consensus for change, building capacity, supporting to policy dialogue, informing new laws and regulations, and creating or improving institutions—was observed almost equally frequently, in about half of the knowledge activities reviewed. Support to Bank strategy formulation or to lending operations was a much less frequent use of knowledge—as expected in view of the overall limited lending in these countries. The overwhelming majority of IFC Advisory Service projects aimed to raise stakeholder awareness (f). This is crucial in projects that aim to introduce new practices, such as the use of credit bureaus, or influence the design of certain reforms, such as reducing the cost of real estate transactions or tax compliance costs for SME. Supporting policy dialogue and informing new laws and regulations were the next two major uses. Figure 2.1. Use of World Bank Knowledge Services and IFC Advisory Services (in % of Reviewed Activities) (A) WORLD BANK KNOWLEDGE ACTIVITIES (B) IFC ADVISORY SERVICES Raise stakeholder awareness Raise stakeholder awareness Design or reform government programs Support dialogue in a sector/on a topic Help reach consensus for change Inform new laws or regulations Build recipient capacity Build recipient capacity Support dialogue in a sector/on a topic Design or reform government programs Partly Partly Inform new laws or regulations Yes Create or improve institutions Yes Create or improve institutions Help reach consensus for change Support Bank strategy formulation Support Bank strategy formulation Support the design of lending operations Support the design of lending operations 0 25 50 75 100 0 25 50 75 100 % % 2.22 Client participation (full or partial) in the Bank Group’s knowledge activities was generally high—an indication of client interest and relevance of the work conducted. Client participation was highest at the scoping stage of Bank knowledge activities and at the review and feedback stages, where it reached almost 80 percent of knowledge activities reviewed by IEG (figure 2.2). Participation was also high at the design and dissemination stages and relatively less frequent during 23 CHAPTER 2 RELEVANCE OF THE WORLD BANK IN KNOWLEDGE-BASED PARTNERSHIPS implementation and analysis of results, reaching about 60 percent. Clients participated in the formulation of recommendations in almost half of the knowledge activities reviewed. Somewhat different from Bank knowledge activities, client participation in IFC Advisory Services was high not only at the design stage but also during implementation (figure 2.2). Discussions of Bank knowledge services with senior policy makers took place in the great majority of activities reviewed by IEG and somewhat less frequently for IFC Advisory Services (table 2.2). 12 The World Bank typically takes the lead in dialogue on policy issues, especially at the senior levels of government, with IFC working at the technical levels. This informal coordination of policy and technical dialogue between the Bank and IFC has worked fairly well in some cases reviewed by IEG in the focus countries—for example, in the work in the Russian Federation on reforming land transactions and subnational Doing Business environment or in the IFC financial sector work in China—but less well elsewhere, as, for example, the work on competition policy and administrative barriers in South Africa. Figure 2.2. Participation of Government Counterparts in Knowledge Activities (in % of Reviewed Projects) (A) WORLD BANK KNOWLEDGE ACTIVITIES (B) IFC ADVISORY SERVICES Review and feedback Design Scoping of activity Scoping of activity Dissemination Implementation Design Review and feedback Partly Partly Implementation Yes Dissemination Yes Analysis of results Analysis of results Formulation of recommendations Formulation of recommendations 0 20 40 60 80 100 0 20 40 60 80 100 % % 2.23 The frequency of full dissemination of Bank knowledge services was relatively low, conducted in slightly more than half of knowledge activities reviewed (table 2.2). In many cases the reports were disseminated internally and externally, with participation of civil society organizations, and served to promote a broader consensus on the reform agenda. A good example is the knowledge activity conducted in Bulgaria. The 2007 Country Economic Memorandum, with a strong focus on education, was disseminated in a session with Parliament, organized jointly 24 CHAPTER 2 RELEVANCE OF THE WORLD BANK IN KNOWLEDGE-BASED PARTNERSHIPS with a think tank. The series of reports on regulatory reform were disseminated through public hearings organized by the Ministry of Economy, with participation of think tanks and professional associations. By contrast, broad dissemination has been a weak point of the Bank engagement in Kazakhstan, perhaps reflecting the authorities’ preference. For example, as brainstorming sessions are confidential, lessons learned or a summary of the discussions are not disseminated to a wider audience, and little if any of the Bank’s knowledge services is being discussed openly with representatives of business associations and other stakeholders. Full dissemination of IFC Advisory Services was more frequent than for Bank knowledge services. The confidentiality often imposed on Bank knowledge services cofinanced by clients or through RAS explains to a large extent the lower rate of dissemination of Bank knowledge services. Table 2.2. Outreach of Bank Group Knowledge Services Percentage Indicators Categories NA No Partly Yes World Bank knowledge 3.6 4.6 7.7 82.7 Product of the activity discussed with senior services policy makers IFC Advisory Services 15.6 12.5 3.1 68.8 World Bank knowledge 2 29.6 11.2 55.6 Product of the activity disseminated and services covered by the media IFC Advisory Services 6.3 18.8 3.1 71.9 Source: Independent Evaluation Group knowledge activity reviews. Note: NA = Not applicable; Some questions on the tasks (such as, Was the activity requested by the client?) were more objective than others (such as, Was the design appropriate to meet objectives?) and accordingly were less subject to observation errors. The indicators in the table average the responses across all tasks and are likely to average out the observation errors of responses on the individual knowledge activity. Strategic Positioning of the World Bank on Development Knowledge 2.24 IEG’s review of the World Bank’s role as a knowledge service provider in the focus countries broadly confirms that the Bank succeeded in strategically repositioning itself as a relevant development partner despite diminishing engagement through finance. Counterparts interviewed by IEG generally acknowledged four strengths in the Bank’s knowledge services that enhance their relevance and help maintain the Bank’s position as a knowledge partner of choice: • Ability of benchmarking against international best practice through cross- country comparisons • Reputation as an independent and credible broker of knowledge with a partnership approach • Knowledge of the local context and capacity to customize international best practice solutions 25 CHAPTER 2 RELEVANCE OF THE WORLD BANK IN KNOWLEDGE-BASED PARTNERSHIPS • Capacity to see the big picture and analyze cross-sectoral issues important for development. 2.25 International benchmarking. In several countries, counterparts were most appreciative of knowledge activities based on standard methodologies developed by the Bank, providing a diagnosis and benchmarking against international best practice. This was the case in Kuwait with regard to the Country Procurement Assessment Review (CPAR), Public Expenditure and Financial Accountability (PEFA), and FSAP reports. The Russian Federation and Kazakhstan particularly appreciated knowledge services that enabled benchmarking of performance with peer country groups—including at the subnational level in the Russian Federation— with use of methodologies, such as ICAs and Doing Business, that were not easily replicable by other partners. Application of these methodologies is generally appreciated on important private sector development issues where there has not been systematic thinking, such as economic diversification, innovation, and competitiveness. Similarly, in Malaysia, the Bank is considered particularly well positioned on economic analysis and knowledge of international experience. Counterparts tend to turn to the Bank for cross-country comparisons, and to consulting firms (Boston Consulting Group, McKinsey) for Malaysia-specific issues. 2.26 Independence and credibility. This attribute of Bank knowledge services came often on top of counterpart views. Perceptions of independence and credibility of high-quality work helped the Bank to succeed in its re-engagement with Malaysia after more than a decade of absence. In Kuwait, counterparts see the World Bank, in comparison to other consulting firms, as an independent broker of knowledge, with high integrity, a partnership approach, high-quality standards, and the capacity to transfer best international practices and experiences. In Chile, an OECD member country, the Bank is still widely recognized as a credible and neutral partner that can be relied upon for good-quality advice. But in South Africa, the Bank has faced more challenging conditions in delivering a knowledge-based country program, due to the unique social political economy of this country. The views within the government on strategy and policies are diverse and reflect different political ideology of different interests. A common criticism has been that some recommendations did not take into account local context, as in the cases of IFC Advisory Service projects with the Department of Trade and Industry. Given the lack of consensus on policy directions, the Bank’s attempt to engage on the policy dialogue as an independent and credible knowledge broker was marginally successful. However, the Bank is still well regarded and it has been relatively more successful in responding to demand-driven Advisory Service requests drawing on its extensive international experience. Perhaps surprisingly, some of the most appreciated Bank contributions in China were on topics of a controversial and 26 CHAPTER 2 RELEVANCE OF THE WORLD BANK IN KNOWLEDGE-BASED PARTNERSHIPS politically-cum-institutionally sensitive nature, such as land policy reform, delivery of rural public services and their financing mechanisms, food safety, and the economic value of scarce water resources. At the same time, the government requested IFC help in improving product quality and corporate governance, and introducing corporate and social responsibility guidelines to private sector and state firms. As long as the Bank takes a holistic, fundamental, and nonconfrontational approach and uses evidence-based—as opposed to ideology-driven—analysis, counterparts in China felt that they were confident enough to handle tough messages. 2.27 Customization to local context. A distinct strength of Bank knowledge services mentioned by counterparts interviewed by IEG is the Bank’s deep knowledge of the local context. As underlined, for example, by stakeholders in Kazakhstan, this knowledge is reinforced by the presence of the Regional Office for Central Asia in Almaty. IFC Advisory Service projects in the Russian Federation have used a combination of local working groups with international experts in formulating recommendations. In Thailand, unlike other development agencies, the Bank is seen by counterparts as having the capacity to properly customize international best practice to the Thai context because of its knowledge of local institutions that comes mainly from the expertise of staff in the Regional Country Office in Bangkok. 2.28 However, the Bank’s recognized capacity to customize knowledge services to the local context might, to some extent, be challenged in the future by the lack of sufficient knowledge activities to gain appropriate depth and breadth of country knowledge. This is a risk mainly when the Bank works through RAS and does not maintain local presence. In Chile, for example, where the program is managed from the Regional Office in Peru, the Bank has not conducted a comprehensive assessment of development priorities since the 2006 Development Policy Review and does not conduct Public Expenditure Reviews or similar cross-cutting studies. 13 The Bank participated in an FSAP update led by the International Monetary Fund (IMF) in FY11. Country visits are being conducted occasionally, to gain perspective on macroeconomic developments and structural policy issues. However, it is questionable whether this is enough for the Bank to maintain a solid analytical base and policy expertise on the country. This might create a gap in the practical relevance of the Bank’s knowledge services, as already seen in some cases. For example, work on the decentralization of education and health validated the authorities’ views on the subject but lacked some practical aspects for implementation. Similarly, some of the work on improving public sector management was conceptually sound but did not refer enough to local Chilean processes to be of practical implementation value. 27 CHAPTER 2 RELEVANCE OF THE WORLD BANK IN KNOWLEDGE-BASED PARTNERSHIPS 2.29 Capacity to see the big picture and deliver multisector development solutions. A strength of the Bank’s knowledge services often emphasized by counterparts, as, for example, in the Russian Federation, was the ability to see the big picture and provide credible advice on the role of government versus the private sector on topics such as economic diversification and innovation. Global consulting firms are seen as quite good in providing narrow technical advice but have less credibility on policy matters. In China too, the Bank enjoys a special advantage in cross-sectoral activities. This advantage stems in part from the Bank’s wide exposure to a variety of sectors, through lending or knowledge services, and in part from the government’s difficulty to assign cross-sectoral mandates in China’s administrative system. The ongoing road safety knowledge product illustrates this. Thanks to the Bank’s expertise and project experience in the various areas—health, road design and services, and others—it brought together sectors that until now were disjointed from the point of view of road safety. 2.30 Again, this advantage might be challenged by the growing tendency of the Bank to deliver knowledge services through the “consultant firm model,” with insufficient follow up and emphasis on important issues for the development agenda. This has been the trend of engagement, for example, in Chile where the Bank works certainly in a demand-driven orientation but sometimes in a piecemeal manner, with insufficient follow up and with insufficient emphasis over time on issues that have been identified as important for medium-term development. Examples include the work on innovation policy where the Bank’s work was virtually suspended following the change in government priorities and the cancellation of the Innovation and Competitiveness investment loan. The programmatic TA on poverty and equality of opportunity was initially used as an input for discussion in the Presidential Council. However, contrary to plans, the work on assessing inequality of opportunity in Chile was discontinued because of the change of priorities of the new government. Moreover, the Bank’s self-imposed boundaries of intervention often keep it out of debates on the long-term development agenda. For example, the Bank is not part of the debate on the all- important long-term issues of labor productivity, innovation, and climate change. Counterparts outside of the government miss the Bank’s contributions and participation in these areas. 2.31 The Bank’s main strengths in knowledge, as perceived by stakeholders in the focus countries, are intimately linked to the Bank’s roles as “knowledge producer” and “knowledge customizer.” They are also linked to its role as “knowledge connector,” as the Bank’s convening power was often used in the focus countries to mobilize top international experts for brainstorming sessions and seminars with high-level government officials (as, for example, in Kazakhstan and Malaysia), or for 28 CHAPTER 2 RELEVANCE OF THE WORLD BANK IN KNOWLEDGE-BASED PARTNERSHIPS TA and working sessions with government agencies to address more specific issues (as, for example, seminars on transfer pricing or on anticorruption organized in Thailand). 14 The results of client surveys in four of the focus countries are broadly consistent with the analysis in this report, underscoring the importance of sharing knowledge about international best practices and of the technical quality of knowledge services (see box 2.2.) Box 2.2. Perception of Knowledge Services from Client Surveys The Bank systematically conducts Country Opinion Surveys in client countries to enhance understanding of how stakeholders perceive Bank work. Country Opinion Surveys are available for four of the focus countries: Bulgaria (FY12), Chile (FY10), China (FY12), and South Africa (FY12). A reading of the section on knowledge indicates that views expressed by stakeholders surveyed were broadly consistent with the analysis in this report. • In Bulgaria, over half of the respondents indicated that the Bank’s greatest value was its knowledge and advisory services. The greatest weakness respondents (23 percent) found was that there was not enough disclosure of the Bank’s work, and that in some instances, the Bank tended to impose technocratic solutions without regard to political realities (22 percent). When asked how the Bank could be of greater value, more than 40 percent of respondents indicated that it should offer more innovative knowledge services. • The greatest value perceived by respondents in Chile (52 percent) was the Bank’s technical advice, and producing knowledge and research that are useful (44 percent). The ability to adapt its knowledge to a country’s needs received a relatively lower rating than other ratings related to the Bank’s knowledge services, and over half of respondents indicated that the Bank’s greatest weakness was the little effort devoted to disseminate its analytical work and policy proposals. A number of respondents reported that the key way to increase the Bank’s value was by adding more value to the internal policy debate through its international experience, and technical assistance. • In China more than 40 percent of surveyed stakeholders felt that the greatest value provided by the Bank was the transfer of international best practices and new concepts and innovations. According to 30 percent of those surveyed, the greatest Bank weakness is that it fails to adapt global knowledge to local conditions. Nearly 40 percent of respondents indicated that in future the Bank—to be more valuable to China—should improve the quality of its experts as related to China’s specific challenges, while 30 percent said the Bank should offer more innovative knowledge services. • In South Africa—where there was a very low stakeholder response (14 percent) and which showed the lowest scores of the four client surveys—technical assistance was seen by respondents as an important value the Bank brought, although financial resources were seen as the item bringing the most value. The weaknesses identified by respondents were not enough public disclosure of Bank work, and Bank staff not being accessible enough. Respondents saw scope to improve value through more responsiveness and staff accessibility, and by having experts who are familiar with South Africa’s specific development challenges. The low response rate may indicate that responses are not representative of the stakeholder sample. Source: Latest country surveys for Bulgaria, Chile, China, and South Africa. 29 CHAPTER 2 RELEVANCE OF THE WORLD BANK IN KNOWLEDGE-BASED PARTNERSHIPS 2.32 There is room for further enhancing the Bank’s relevance as knowledge connector by intermediating more efficiently the knowledge acquired in countries with knowledge-based programs. There were ample opportunities for learning from development experiences in the focus countries. For example, the engagement in Chile generates highly relevant knowledge for other countries in view of Chile’s progress in several areas of public policy (education financing, program budgeting, pensions, insurance) and strong institutional capabilities (for example, on the evaluation of policy programs). Malaysia’s development trajectory from low-income to upper-middle-income economy, and its central place in the Association of Southeast Asian nations (ASEAN) community, holds relevant knowledge for other countries. Counterparts interviewed by IEG saw the Bank as a platform to share Malaysia’s experiences with other developing countries. Malaysia has offered to host regional experience sharing events in the past (regional conference on investment climate), although evidence on the effectiveness of this activity was difficult to come by. In Kazakhstan, the findings and recommendations of some knowledge activities, such as study on oil funds and the sovereign wealth fund, could be useful to other countries. In Kuwait, the program generated knowledge, such as the analysis in policy notes on the governance and integrity program, which could be relevant to other countries in the Middle East and North Africa. As China—and other upper-middle-income countries—becomes an increasingly important provider of global development assistance, greater understanding of its successes and failures, especially with a global perspective, will help other countries assess the relevance of the Chinese model to their own development challenges. In particular, issues related to land and water scarcity—tenure security, registration and cadaster, productivity of land—are concerns worldwide, especially in the Middle East and North Africa and much of Sub-Saharan Africa, and the Bank could play a useful role in facilitating exchanges based on the knowledge acquired in China. 2.33 There are some good examples where the Bank’s knowledge services facilitated South-South exchanges and policy dialogue in the focus countries (see chapter 4), but more can be done as the Bank’s geographic, thematic, and organizational fragmentation prevents the full potential of such exchanges to be realized. In China, for example, while the World Bank Institute’s technical assistance work has focused squarely on catalyzing lessons for other developing countries, it appears that the Bank has not fully exploited the potential of this mutual learning opportunity. The Russian Federation, for example, could benefit from the extensive work the Bank has done in China on regional approaches to investment promotion. Lessons from this experience have not been transmitted to the Russian Federation. 30 CHAPTER 2 RELEVANCE OF THE WORLD BANK IN KNOWLEDGE-BASED PARTNERSHIPS Knowledge sharing is often hampered by the confidentiality of the work conducted through RAS. In Kuwait, there has been no transfer of knowledge to other countries given the confidentially of the studies. Intermediation of knowledge often takes place only informally, through Bank staff working also on other countries or through South-South exchanges initiated by clients and sometimes facilitated by the Bank. In Chile, the Bank has used mostly its informal networks, through the task team leaders and network management, to convey knowledge acquired in Chile to other countries in Latin America and elsewhere. In Kazakhstan, some government agencies have already shared their experiences with other countries in the region. For example, the Statistical Agency is sharing its experience with Azerbaijan and other countries in the region without World Bank intervention. 2.34 The strengths and weaknesses of the Bank’s knowledge services in the focus countries determine, to a large extent, the way the Bank is perceived by clients in its positioning as a development partner with respect to other institutions—other international financial institutions, universities, think tanks, and global consulting firms. It should be noted that, in keeping with the Bank’s mandate as an international development institution and its policy statements, “strategic positioning” is meant here to illustrate how the Bank is perceived as a strategic partner and policy advisor, not whether the Bank could position itself as a consultancy firm competing for advisory contracts, which is irrelevant to its mission. 15 Based on IEG interviews of stakeholders, the Bank’s strategic positioning with respect to other knowledge providers is generally favorable, but varies across countries and thematic areas of work. The Bank’s strategic positioning also depends on the mode of knowledge services financing as the presence of other knowledge service providers is more intense when the Bank’s knowledge services are being delivered as a RAS than as a public good. A summary of the World Bank’s perceived strategic positioning in the focus countries is in Appendix G. 2.35 The strategic positioning of IFC Advisory Services in focus countries is based on institutional or practice gaps (or market failure) in areas relevant to the CPS, and IFC uses the “additionality” 16 of Advisory Services to enable the implementation of reforms to address these gaps. There is intensive awareness-raising and dissemination of IFC work as well as piloting. Results are based on government action, irrespective of whether the subsequent TA is delivered by IFC or others. However, IFC does not have the resources to provide TA, for example, to all provinces in China or support all entrepreneurship programs in South Africa. In a sense, IFC opens up the market for others, in the same way that IFC investments are meant to encourage other private sector investments. IFC has a poverty focus, with emphasis on IDA and fragile states, and it would thus typically reallocate resources to poor countries if Advisory Services have no impact in upper-middle-income 31 CHAPTER 2 RELEVANCE OF THE WORLD BANK IN KNOWLEDGE-BASED PARTNERSHIPS countries with World Bank knowledge-based programs. IFC continues to work in some larger middle-income countries—such as China and Russia—with an emphasis on public goods or frontier areas with a disproportionate lower-income population. 2.36 Overall, despite variation across countries and sectors, the Bank has succeeded in capitalizing on some key strengths (international experience, credibility and independence, customization, multisector expertise) so as to remain relevant and maintain a good strategic position in the countries where it engages mainly through knowledge-based programs. Risks that may erode these strengths in the future would merit attention. They reflect in particular the possible loss of breadth and depth in analytical knowledge of country clients, and the insufficient focus on medium-term country development challenges as a result of increasing concentration on RAS in high client demand. The Bank’s overall relevance could be further enhanced by improving the way the Bank intermediates the knowledge acquired in country clients with knowledge-based programs. 32 3. Technical Quality of Knowledge Services Highlights  The majority of Bank knowledge services and International Finance Corporation (IFC) Advisory Services conveyed to the client international best practice and relevant examples, based on the Bank Group’s global knowledge. Most tasks referred to the institutional and policy context and, to a somewhat lesser extent, generated some kind of new evidence and provided actionable recommendations.  There was more variation in the degree to which the tasks used local expertise and discussed capacity requirements of recommendations.  About 48 percent of Bank knowledge activities did not deploy, or only partly deployed, local expertise. Use of local expertise was more frequent in work on education and health, agriculture, and poverty and social protection.  Actionable recommendations were formulated in 69 percent of Bank knowledge activities but their substance and attention to capacity requirements varied considerably across tasks.  The capacity requirements and administrative feasibility of recommendations were discussed less frequently and were neglected in 25 percent of Bank knowledge services. They were addressed more frequently by activities in the financial sector and infrastructure. 3.1 This chapter reviews how the selected Bank Group knowledge services performed, as captured by three input factors: the transfer of global expertise, the customization of solutions to country context, and the development of actionable recommendations. Leveraging the Bank’s Global Expertise 3.2 It is, of course, difficult to compare the technical quality of heterogeneous knowledge services. But an indication of quality can be gathered from the knowledge inputs that went into the service. Inputs include deployment of global knowledge, consideration of the institutional and policy context, production of new evidence, development of recommendations, and consideration of capacity constraints for implementing recommendations. The great majority of Bank knowledge services and International Finance Corporation (IFC) Advisory Services conveyed to the client international best practice and relevant examples from the region or other regions, based on the Bank Group’s global knowledge (table 3.1). A look at some examples covered in this study illustrates more specifically the myriad 33 CHAPTER 3 TECHNICAL QUALITY OF KNOWLEDGE SERVICES ways deployment of international experience can contribute to technical quality and to client use of the activity (for example, on regulations, programs, and capacity). Global knowledge can help benchmark existing institutions and policies, and guide reform toward best practices in these areas. In the focus countries, the Bank used very different knowledge vehicles, from broad-ranging reports to more narrowly focused advisory and custom-made technical assistance (TA), to convey global knowledge and international best practice. Successful tasks on the deployment of global knowledge ranged from broad to very specific applications. Table 3.1. Transfer of Global Knowledge in Knowledge Activities and Advisory Services Reviewed by IEG Indicators Categories Percentage NA No Partly Yes World Bank Knowledge 2.6 4.6 14.8 78.1 Transfer of international best practice and relevant Services examples IFC Advisory Services 15.6 0 21.9 62.5 Source: Independent Evaluation Group (IEG) knowledge activity reviews. Note: NA = Not applicable. Some questions on the tasks (such as, Was the activity requested by the client?) were more objective than others (such as, Was the design appropriate to meet objectives?) and accordingly were less subject to observation errors. The indicators in the table average the responses across all tasks and are likely to average out the observation errors of responses on the individual knowledge activity. 3.3 At the broadest level, an example of a successful task was the Collaboration Forum for a Stronger Thailand that brought together international experts in a wide range of development topics. It helped inform new government programs on a rice disaster insurance scheme, increase attention to debt sustainability (with follow-up staff training from the Bank on public debt management modeling techniques), and introduce new concepts on financing public services, including ways to use public- private partnerships. The Kazakhstan Growth and Competitiveness—Issues and Priorities in the areas of Macroeconomic, Industrial, Trade, and Institution Development Policies activity also tackled a broad set of issues, and provided many examples of best practices as well as practices to be avoided, and relevant policy examples from other oil-rich countries. 17 3.4 Mostly, however, Bank knowledge products focused on specific issues in the multiple sectors that the Bank covers. In much of its knowledge work, the Bank provided advice developed in a largely custom-made fashion to address specific client needs. A particularly successful example was the support for education policy in Chile, which brought in institutional experience from a set of good performing educational systems relevant to Chile (see appendix H). The South Africa Rural Development and Land Reform Reimbursable Technical Assistance conveyed global knowledge in both of these areas through its international consultants as well as seminars and study tours; these likely contributed to the many uses of the task 34 CHAPTER 3 TECHNICAL QUALITY OF KNOWLEDGE SERVICES including, for example, suggestions for improving and simplifying the cadaster system. China Air Pollution Control brought in advanced country knowledge on management of fine particles; it helped achieve a consensus to address the matter and expand monitoring and control plans. A hospital autonomy report under the Thailand Health Nutrition and Population knowledge activity brought in lessons from elsewhere to inform Thailand’s difficult policy dialogue on decentralization. The Chile Implementation of Risk-Based Supervision Model for Insurance Industry brought in experience from Canada that informed new laws and regulations. In the Kazakhstan Poverty Monitoring task, the team shared a number of good international practices on survey instruments, and a study tour under the task conveyed international experience in data collection and dissemination at the U.S. Census Bureau in Washington. Work on innovation management in the Russian Federation focused primarily on international practices to provide advice to the Russia Venture Company on options for network models in supporting innovation; the company used its output to decide on the best network model for innovation programs. The Kazakhstan Higher Education TA brought in best practices by supporting a study tour of leading universities in the world, and then holding a seminar to review what was learned on the study tour and discussing a revised plan for the new university the government envisaged, with feedback provided by the universities visited. The TA may have helped establish a more autonomous university and initiated efforts to benefit from an international advisory panel of experts. 3.5 A number of knowledge activities brought in global knowledge through more standardized techniques and methods that rank high in client appreciation (see chapter 2), including Reports on the Observance of Standards and Codes (ROSCs), Financial Sector Assessment Programs(FSAPs), and Investment Climate Assessments (ICAs). The China Report on the Observance of Standards and Codes, which brought experience from Hong Kong SAR, China, Malaysia, Singapore, and Thailand, helped raise awareness of the shortcomings in the accounting industry and the urgent need to produce more high-quality corporate accounts and auditors to enable preparation of credible financial statements. Similarly, the Bulgaria Corporate Governance TA, which included a ROSC, a number of seminars, and other IFC Advisory Services for strengthening the Sofia stock exchange and increasing the number of corporations, brought in useful international examples and joint seminars that helped Bulgarian firms increase awareness and develop capital market regulation. The Russian Federation FSAP Update followed a methodology in line with international best practice and included a detailed assessment of the International Organization of Securities Commission principles of regulation, stress testing of the system, and an assessment of Basel Core Principles. The Central Bank widely used it as a supporting source of information and reference that added weight to its reform proposals. The Thailand Investment Climate Assessment activities provided international benchmarking that raised awareness and interest in 35 CHAPTER 3 TECHNICAL QUALITY OF KNOWLEDGE SERVICES investment climate policies and encouraged further work. The Malaysia Productivity and Investment Climate Assessment Update also provided international benchmarking, but its impact on awareness was probably weak. 3.6 Several IFC Advisory Service projects focused on disseminating global best practices as part of awareness-raising initiatives to generate interest in and ownership of reforms that are yet to be defined. In China, IFC Advisory Services on housing finance, leasing, and personal bankruptcy legislations were meant to expose authorities to global practices. The Advisory Services on secured transactions had several components, the first of which included exposure of stakeholders to best practices. In work on credit bureaus, IFC arranged seminars that included leading global players, shared best practice regulatory guidelines from other countries that were translated into Chinese, and facilitated the participation of Chinese officials in the World Consumer Credit Reporting Conference. In the Russian Federation, the work on introducing legislation that would enable emergence of securitization transactions involved reviewing best practice legal environments, while the outputs of the Advisory Service project on primary mortgage included a mortgage lending manual and mortgage borrower consumer protection regulation based on international best practice. 3.7 In a number of cases, however, the Bank did not deploy the global knowledge the tasks would have warranted. The China Private Sector Development Program, which called for advice on innovation systems, is one example. Similarly, there is no record that the Kazakhstan’s Public Administration Reform or its Insolvency Framework—A Preliminary Overview delivered the global knowledge expected of them. In a few other cases, global knowledge appeared to be less critical for the task at hand, although they were likely to provide opportunities to bring in international experience. Examples are the Russian Federation Agricultural Policy Monitoring TA activity, which financed donor coordination efforts by the Bank to curtail overlap and contradictory advice from donors, and the China IFC Advisory Services— Access to Finance, which focused on reviewing the domestic situation in China by establishing baseline information through a survey that would be performed periodically. 18 Customization of Solutions to Country Context 3.8 Bringing in global knowledge that is relevant to a country, as in the Chile Primary and Secondary Education knowledge activity (see appendix H) and the other examples above, is an important step in customizing to the country’s context. It involves choosing the right comparators and the global practices that are appropriate for the country context. Accordingly, customization begins with reference to the institutional and policy context in the country. Furthermore, it may 36 CHAPTER 3 TECHNICAL QUALITY OF KNOWLEDGE SERVICES need generation of new evidence or data that inform analysis and recommendations. And both of these may also benefit from involvement of local expertise that is familiar with country conditions and data. 3.9 Most tasks generated some information and, not surprisingly, virtually all tasks covered by this study considered the country’s institutional and policy context in one way or another (table 3.2). In contrast, about 48 percent of Bank knowledge activities did not deploy (or only partly deployed) local expertise. This may reflect unavailability of such expertise in some countries or sectors, a constraint on quality in such cases. Use of local expertise was frequent in work on education and health, agriculture, and poverty and social protection (figure 3.1). It was rare in work on the financial sector, where perhaps unavailability of expertise may be more binding, but was also infrequent in work on economic policy and public sector governance. Among the evaluation’s focus countries, local expertise was used more frequently in Bulgaria, China, and South Africa, and less frequently in Kazakhstan, Kuwait, and the Russian Federation (figure 3.1). A look at the substance in the tasks indicates that attention to country policies and institutions varied in depth. Use of information ranged from reliance on existing data to conducting surveys that produced fresh new data. And use of local expertise varied from little or none to close collaboration in actual production of knowledge activity outputs. Table 3.2. Customization to Local Context in Knowledge Activities and Advisory Services Reviewed by IEG Indicators Categories NA No Partly Yes World Bank 1.5 1.5 14.3 82.1 Reference to the institutional and policy context in Knowledge Activities the country IFC Advisory 12.5 6.3 6.3 75 Services World Bank 2 10.2 17.3 69.9 Generated new evidence and data that inform Knowledge Activities policy making IFC Advisory 28.1 6.3 9.4 56.3 Services World Bank 3.6 31.1 16.8 48 Knowledge Activities Use of local expertise IFC Advisory 6.3 28.1 12.5 53.1 Services Source: Independent Evaluation Group (IEG) knowledge activity reviews. Note: NA= Not applicable; Some questions on the tasks (such as, Was the activity requested by the client?) were more objective than others (such as, Was the design appropriate to meet objectives?) and accordingly were less subject to observation errors. The indicators in the table average the responses across all tasks and are likely to average out the observation errors of responses on the individual knowledge activity. 37 CHAPTER 3 TECHNICAL QUALITY OF KNOWLEDGE SERVICES Figure 3.1. Use of Local Expertise in World Bank Knowledge Activities (in % of Activities Reviewed by IEG) (A) THEMATIC AREAS (B) FOCUS COUNTRIES Education and Health South Africa China Agriculture and Rural Development Bulgaria Poverty Reduction and Social Protection Chile Infrastructure, Urban Development and … Malaysia Private Sector Thailand Russian Federation Economic Policy and Public Sector Governance Kuwait Financial Sector Development and Financial … Kazakhstan 0 10 20 30 40 50 60 70 80 90 100 0 20 40 60 80 100 % % 3.10 Thailand’s economics of AIDS report (see appendix H) provides a good example of coupling strong deployment of international experience with thorough consideration of the institutional and policy context, production of key information (first estimates of cost effectiveness of interventions) for decision making on HIV/AIDS policy, and reliance on local health expertise. The other examples that follow illustrate the wide range of variation in the extent to which tasks considered the country context, produced new information, and engaged local expertise. 3.11 The South Africa Private Sector Development Activities (Small, Micro, and Medium Enterprises) was fairly strong in the three areas. It considered the country context, including economic trends, the role of small and medium enterprises, and industrial policies. It conducted a micro-enterprise survey that used the same core questionnaire of an Investment Climate Assessment—which would provide consistency for benchmarking—and added questions on the incentives framework; the survey was implemented using local expertise. The result was a credible report that was a major input to the government micro, small, and medium enterprises strategy. The Bulgaria Railways Policy Note is an example of thorough coverage of country context, use of existing data, and moderate use of local expertise. It was substantiated by a technical report that covered industry structure, markets, and operational and financial performance, with reference to international benchmarks, compilation of data from several existing sources, and participation by counterparts in providing access to data and providing comments on drafts. In China, a capital market development project is an example of detailed coverage of the institutional and policy context and reliance on a local drafting team of experts to draft the report in Chinese, while using existing data. It contributed to capacity building at the 38 CHAPTER 3 TECHNICAL QUALITY OF KNOWLEDGE SERVICES Research Center of the China Securities Regulatory Commission and to raising the profile of commission and enhancing its role as the regulator of capital markets in China. 3.12 The China IFC Advisory Services on secured transactions and credit bureaus used programmatic approaches to expose stakeholders to best practices as part of the awareness-raising phase, followed by design discussion on how to blend best practices with local context, implementation of pilots to test customized solutions, and scaling up based on lessons from previous phases of the program. In the case of the Russian Federation, the IFC Advisory Services on securitization used a local working group and an international consultant providing inputs on international best practices, to ensure solutions that are feasible in light of the state of local institutions. The primary mortgage Advisory Services project also used a working group that brought together key market players to develop solutions that integrated best international practices and local context. 3.13 A number of reports were rather thin in the three areas of customization to country context. Outputs from the Russia Regional Development Strategy TA, including participation in international conferences, dissemination of the 2009 World Development Report, and a paper on intergovernmental relationships delved on country context and were used only to a very limited extent. A Chile Policy Assessment was likewise limited in its consideration of the country context and did not use local expertise or produce new data. There are no indications that it was used in any way by the government. Development of Actionable Recommendations 3.14 A robust test of the quality or usefulness of a task is its ability to generate consistent and actionable recommendations with a foothold in the country’s institutional reality and its capacity to implement the recommendations. Prioritization of recommendations will also contribute to the usefulness of knowledge activity outputs. Development of recommendations varied considerably across tasks in substance and in attention to capacity requirements. Actionable recommendations were formulated in 69 percent of Bank knowledge services. IFC Advisory Services were more often directed to demonstrate the feasibility of particular solutions to issues of interest to the client rather than formulating recommendations. The capacity requirements and administrative feasibility of these recommendations were discussed less frequently and were neglected in 25 percent of Bank knowledge services (table 3.3). Capacity requirements and administrative feasibility of recommendations was addressed more frequently by knowledge services in the financial sector and in infrastructure, and much less frequently by tasks in poverty and social protection and education and health. Among the focus 39 CHAPTER 3 TECHNICAL QUALITY OF KNOWLEDGE SERVICES countries, these institutional aspects of the recommendations were discussed more frequently by work conducted in China and Thailand and less frequently in Bulgaria, Chile, and the Russian Federation (figure 3.2). Table 3.3. Development of Recommendations in Knowledge Services and Advisory Services Reviewed by IEG Percentage Indicators Categories NA No Partly Yes World Bank Knowledge 3.1 6.6 20.9 69.4 Actionable recommendations formulated consistent Services with findings IFC Advisory Services 34.4 6.3 15.6 43.8 World Bank Knowledge 6.6 25.5 25 42.9 Analysis of capacity requirements and administrative Services feasibility of recommendations IFC Advisory Services 43.8 21.9 9.4 25 Source: Independent Evaluation Group knowledge activity reviews. Note: NA = Not applicable; Some questions on the tasks (such as, Was the activity requested by the client?) were more objective than others (such as, Was the design appropriate to meet objectives?) and accordingly were less subject to observation errors. The indicators in the table average the responses across all tasks and are likely to average out the observation errors of responses on the individual knowledge activity. Figure 3.2. Discussion of Capacity Requirements and Administrative Feasibility of Recommendations in World Bank Knowledge Services (in % of Activities Reviewed by IEG) (A) THEMATIC AREAS (B) FOCUS COUNTRIES Financial Sector Development and Financial Management China Thailand Infrastructure, Urban development and Environment Malaysia Private Sector South Africa Economic Policy and Public Sector Governance Kuwait Agriculture and Rural development Kazakhstan Bulgaria Education and Health Chile Poverty Reduction and Social Protection Russian Federation 0 10 20 30 40 50 0 10 20 30 40 50 60 % % 3.15 The South Africa Sector Study of the Effective Tax Burden (described in appendix H) provides an example of a task with recommendations that were consistent with the analysis (in this case with the calculation of marginal effective tax rates), detailed enough to be actionable, and with a design that considered to some extent the administrative challenges of tax policy implementation. It recommended, for example, a declining-balance approach to depreciation 40 CHAPTER 3 TECHNICAL QUALITY OF KNOWLEDGE SERVICES allowances to improve administration and compliance. The other examples that follow below illustrate the wide variation across tasks in these dimensions. 3.16 The China Rural Public Services knowledge activity included a report aiming to improve the efficiency of capital expenditures with specific recommendations and attention to capacity that made them actionable. 19 Furthermore, the activity supported pilots to test the recommended institutional changes (for example, the use of block grants and performance-based grants) and the training programs to create institutional capacity. The Kuwait National Strategy for the Clean Development Mechanism Reimbursable Technical Assistance included a full report developing recommendations on institutional development. The task sought to provide advice on clean air investments and examined organizational options for the Designated National Authority to conform to the modalities and procedures of the Clean Development Mechanism, a credit scheme under the Kyoto Protocol to implement emission-reduction projects in developing countries. The TA considered five models based on international experience and settled on a single government department option. The recommendations spelled out the structure of the Designated National Authority (board, secretariat, and technical group) and the function of its units; the project approval procedures; and the sustainable development criteria to be met by projects, staffing, and capacity building needs. They clearly covered the “what” and the “how” that recommendations need to be of use. The Russia Programmatic Poverty Work (Stage 2) developed specific recommendations to improve the targeting of social expenditures, consistent with its finding that only a small fraction of noncontributory resources was targeted to the poor and the few income-tested programs explicitly designed to serve poor or vulnerable populations had mediocre targeting. 20 Despite the well-considered recommendations, this knowledge service was not used. It is an example of a task with good overall technical quality, but that suffered from weak engagement of local stakeholders. The report Accelerating Bulgaria’s Convergence: The Challenge of Raising Productivity provides a good example of attention to prioritizing recommendations by defining the top short-term and top medium-term reforms. 3.17 The recommendations developed in three of the knowledge services discussed above contributed to their use for different purposes. In contrast, a number of tasks suffered from weak attention to developing proposals and this contributed to their being used very little. A few examples will suffice. One is the South Africa Administrative Barriers IFC Advisory task, which did not flesh out its recommendations, with its quality also suffering from weak attention to country context. Similarly, in the South Africa Analysis of Investment Climate Variables, recommendations were general, bereft of implementation guidance, and with no discussion of capacity constraints or challenges. Neither of these studies was used to any significant extent. 41 CHAPTER 3 TECHNICAL QUALITY OF KNOWLEDGE SERVICES 3.18 A few tasks were actually not designed to render recommendations. An example is the Basel III Financial Architecture and Emerging Regulatory Developments in Macro Prudential Tools (Russian Federation), which did not have recommendations and aimed simply to create awareness of the changes in Basel III. It probably served to confirm the client’s own conclusions and vision in relation to the impact of Basel III on the Russian banking system. Without recommendations, its use was commensurately limited. 42 4. Results and Their Sustainability Highlights  Expected outcomes were achieved—or likely to be achieved—in 47 percent of Bank knowledge services and were partly achieved in another 37 percent. The frequency of outcome achievement was broadly comparable for Bank economic and sector work (ESW) and technical assistance.  Outcomes of International Finance Corporation (IFC) Advisory Services were achieved in about 38 percent of projects reviewed.  In general, knowledge service outcomes were more often achieved when the focus of the work was in relatively narrower thematic areas, rather than in broader themes, encompassing an ambitious reform agenda, or when the achievement of results would necessitate multisector efforts.  The achievement of knowledge service outcomes was not correlated with financing arrangements for Bank knowledge services, probably owing to the high relevance of Bank knowledge services in countries with knowledge-based programs.  Even though lending was limited, it remained a powerful driver of results for the Bank’s knowledge services as at least partial achievement of expected outcomes of Bank knowledge services was observed more frequently when knowledge activities were used for the design of lending operations.  Knowledge services with outcomes likely to be achieved more frequently had designs appropriate to meet their objectives, were requested by the client, and were able to address issues relevant to the client compared to knowledge services with outcomes unlikely to be achieved.  Knowledge services that lagged in the achievement of outcomes were also weak in conveying international best practice, providing relevant examples, producing new evidence useful for policy making, formulating actionable recommendations, and discussing the capacity requirements and administrative feasibility of the recommendations.  The lack of timely delivery of knowledge services to affect important decisions—not a prevalent problem in the sample of countries—was associated with poorer outcomes.  Client participation in the various stages of knowledge services and participation of local experts were associated with the achievement of results.  Knowledge services that achieved results have more often contributed to strengthening institutions and in building analytical and policy formulation capacity of recipients. This also contributed to the sustainability of these outcomes.  Monitoring of Bank knowledge service results was weak for both individual activities and the country partnership strategy (CPS). Only 23 percent of knowledge activities included, at least partly, results indicators to track the achievement of the activity’s outcomes. The majority of IFC Advisory Services were at least partly equipped with results indicators.  Factors that influenced the degree to which World Bank and IFC knowledge services had synergy included the quality of the CPS results framework and the existence of core ESW. There is scope for improving coordination 43 CHAPTER 4 RESULTS AND THEIR SUSTAINABILITY between World Bank and IFC, for example by including one institution in the other’s review processes, especially at the concept and design stage. 4.1 This chapter first evaluates whether the expected outcomes of Bank Group knowledge services in the focus countries have been, or are likely to be, achieved. Next, it reviews the factors associated with the achievement of results of World Bank knowledge services. Third, it focuses on the sustainability of these results and the factors that may affect it. Finally, it reviews the synergy between World Bank knowledge services and International Finance Corporation (IFC) Advisory Services in promoting private and financial sector development in the focus countries. Results of World Bank Group Knowledge Services 4.2 IEG’s review reveals a mixed, though broadly positive picture for the achievement of results by knowledge services in the focus countries. Expected outcomes were achieved (or likely to be achieved) in 47 percent of Bank knowledge services (table 4.1). They were partly achieved in another 37 percent of knowledge activities reviewed. The frequency of outcome achievement was broadly comparable for Bank economic and sector work (ESW) and technical assistance (TA). Outcomes of IFC’s Advisory Services were achieved in about 38 percent of the reviewed projects, lower than for World Bank knowledge services, while the frequency of projects that did not achieve their objectives surpassed that of Bank knowledge products. A first caveat applies to this assessment as the expected outcomes of Bank knowledge services are often not explicitly described and, even less, quantified in formal documents. Thus, expected outcomes often had to be inferred from the activities’ design and overall stated purposes as well as from interviews with stakeholders and staff. A second caveat is that the results of knowledge services may potentially require a long time before they materialize, as the gestation period of difficult reforms may be long and capacity building or institution strengthening typically require significant time. Moreover, the Bank’s knowledge services are often used to raise awareness or inform internal government debates, with the results appearing with substantial time lags. The Independent Evaluation Group’s (IEG’s) assessment of the achievement of knowledge service outcomes in the focus countries is thus surrounded by a degree of uncertainty as the observed achievement may be subject to change over time. 4.3 The assessment of outcomes for a task or group of tasks indicates the extent of progress achieved toward outcomes, including progress through the policy- making process. The assessment was based on the feedback obtained during the country visits and desk reviews and was limited by the generally weak monitoring 44 CHAPTER 4 RESULTS AND THEIR SUSTAINABILITY of Bank knowledge activity results. The assessed progress varied from tasks that had little or no impact on policies (such as Investment Climate Assessments [ICAs] in Thailand), to tasks with recommendations that were being implemented, but with no visible impact on development outcomes thus far (for example, a report on student loans in Chile), and to tasks where development outcomes were already in evidence (for example, a report on inequality in China, see [Box 4.2]). Table 4.1. Achievement of Expected Outcomes of Bank Group Knowledge Services (in % of Activities Reviewed by IEG) Categories Number of Expected outcome of knowledge activity likely to be activities achieved reviewed No Partly Yes World Bank Knowledge 196 12.8 36.7 47.4 Services Of which, ESWa 117 10.3 40.2 47.0 TA 79 16.5 31.7 48.1 IFC Advisory Services 32 21.9 25 37.5 Source: Independent Evaluation Group (IEG) knowledge activity reviews. Note: Some questions on the tasks (such as, Was the activity requested by the client?) were more objective than others (such as, Was the design appropriate to meet objectives?) and accordingly were less subject to observation errors. The indicators in the table average the responses across all tasks and are likely to average out the observation errors of responses on the individual knowledge activity. a. The 118 economic and sector work (ESW) products reviewed by the Independent Evaluation Group (IEG) include 17 activities reviewed as a bundle of ESW and technical assistance (TA) as the latter was closely linked to the ESW. 4.4 With these caveats in mind, the achievement of outcomes of World Bank knowledge services in the focus countries can be compared to the average performance ratings of World Bank lending operations to get some perspective on the relative success of the Bank’s knowledge services when they are delivered as freestanding products. Over FY06-11, 74 percent of completed World Bank investment lending operations evaluated by IEG and about 82 percent of Development Policy Operations evaluated had a moderately satisfactory or higher development outcome rating. 21 On the other hand, the expected outcomes of about 84 percent of Bank knowledge services reviewed by IEG in the focus countries were likely to be at least partly achieved. Achievement of outcomes of Bank knowledge services in these countries—that make only limited use of Bank financing—thus slightly exceeded the average performance rating of Bank lending operations across regions. In the case of IFC, 58 percent of Advisory Services projects that were completed in FY08-10 and evaluated by IEG had ratings of mostly successful and higher ratings for development effectiveness. Using the (at least partial) achievement of expected outcomes as a proxy for overall development effectiveness, IFC Advisory Services projects reviewed in the focus countries with World Bank knowledge-based programs slightly surpassed overall IFC Advisory Services performance. 45 CHAPTER 4 RESULTS AND THEIR SUSTAINABILITY 4.5 The degree of achievement of knowledge service outcomes varies across thematic areas of knowledge work and countries. Achievement (at least partial) of outcomes was higher than average in agriculture and rural development, education and health, and the financial sector. It was about average in infrastructure, urban development and environment, and lower than average in private sector development and in economic policy and public sector governance (Figure 4.1). In general, knowledge activity outcomes were more often achieved when the focus of the work was in relatively narrower thematic areas, such as education and health, the financial sector, and subsectors in infrastructure. In broader thematic areas, calling for policy action or reform across several sectors, such as private sector development, economic policy, and public sector governance, expected outcomes were achieved less often. A caveat applies to knowledge service outcomes in these broader thematic areas, as the coordination of reforms across sectors may require more time to bear fruit, beyond the period covered by this evaluation. Outcomes were also different across the evaluation’s focus countries. They were at least partly achieved with higher frequency in Bulgaria, China, and Malaysia, and less so in Kuwait (Figure 4.1). A summary of results achieved in each of the focus countries of the evaluation is in Appendix I. Figure 4.1. Achievement of Expected Outcomes of World Bank Knowledge Services by Thematic Area and Focus Countries (in % of Activities Reviewed by IEG) (A) THEMATIC AREAS (B) FOCUS COUNTRIES Agriculture and Rural Development Bulgaria Malaysia Education and Health China Financial Sector Development and Financial Management Kazakhstan Poverty Reduction and Social Protection South Africa Infrastructure, Urban Development and Chile Environment Thailand Private Sector Development Russian Federation Economic Policy and Public Sector Governance Kuwait Outcomes partly achieved 0 20 40 60 80 100 0 20 40 60 80 100 Outcomes achieved Outcomes partly achieved % % Outcomes achieved 4.6 Achievement of outcomes of knowledge activities in poverty reduction and social protection was close to the average across thematic areas, although knowledge activities with partial outcome achievement were relatively frequent (figure 4.1). Despite the strides made by upper-middle countries in lifting millions out of 46 CHAPTER 4 RESULTS AND THEIR SUSTAINABILITY poverty, these countries still face challenges in ensuring the sustainability of poverty reduction and protecting the vulnerable in the face of global economic swings. The Bank continues to be seen as a unique repository of knowledge in poverty reduction and social protection with knowledge activities conducted in almost all of the focus countries of the evaluation but will need to pay focused attention on the factors affecting the results of knowledge services in this area (Box 4.1). Box 4.1. Knowledge Services Results in Poverty Reduction and Social Protection: Stylized Findings Social protection and poverty reduction knowledge activities accounted for about 9 percent of the activities reviewed. Those activities were recorded in seven of the nine countries covered (no Bank knowledge service in social protection or poverty reduction was done in Kuwait or South Africa during the period 2005-2012). Of 20 activities, including programmatic ones, six included technical assistance, either as freestanding interventions (three) or combined with economic and sector work. Thematically, interventions most frequently dealt with employment and labor market issues. Poverty monitoring and old-age security were also prominent, while safety net interventions were few. This may indicate that basic safety net structures are in place, albeit not always well funded, and that the focus has shifted to creating economic opportunity. Stylized findings from the assessment indicate that the focus of the Bank’s knowledge services in social protection and poverty reduction was on supporting the in-country policy dialogue and on providing solutions to specific policy issues as they arose. Over 80 percent of knowledge products fit into this category. Knowledge services did not feature prominently as a springboard for developing Bank strategy or for lending; such was the case in less than 25 percent of activities. In line with the focus on in-country policies, a large share of activities were either wholly initiated by the client or initiated in collaboration with the client; only 15 percent of activities were fully supply-driven. The activities drew heavily on local expertise; this was the case in some 75 percent of cases, largely above the average of knowledge services reviewed, while client participation in various stages of knowledge services ranged from 65 percent to 75 percent. Results frameworks that would have allowed rigorous measurement of outcomes were absent in most instances. Still, all activities were completed, their recommendations discussed with policy makers in 85 percent of cases, and workshops were held, also in 85 percent of cases. Activities were mainly demand-driven, and local expertise, often from government or quasi-government institutions, were involved in most activities. These factors make it likely that the activities did influence policy making and policy design in a sustainable way. The assessment estimated that 80 percent of activities were likely to have at least partly achieved their expected results. 4.7 There are several examples of knowledge activities that have achieved results in the focus countries. Some of these activities generated information necessary for decision making, made a decisive contribution in raising stakeholder awareness, or 47 CHAPTER 4 RESULTS AND THEIR SUSTAINABILITY contributed to strengthening counterpart technical capacity. For example, in China there is evidence that the recommendations of the report Reducing Inequality for Shared Growth in China: Strategy and Policy Options for Guangdong Province, a high- profile study conducted jointly with the provincial authorities, are being gradually implemented with concrete results in declining inequality (box 4.2). The Economics of Effective AIDS Treatment report in Thailand has been useful in providing the needed cost-benefit information to the government to increase the financing of antiretroviral treatment for AIDS patients, while the Harm Reduction Policies and Interventions for Injection Drug Users in Thailand report contains recommendations that are being implemented by nongovernmental organizations, in their HIV harm reduction programs. In education, although the recommendations of the recent skills report in Thailand are considered too general, skill-building is now a top reform priority for the government. Box 4.2. Helping China’s Guangdong Province Tackle Rising Inequality Rising inequality in Guangdong, China’s fastest growing and wealthiest province, is a challenge in view of the government’s goal of achieving a harmonious society. Having decided to tackle head-on the challenge of rising inequality, the provincial authorities requested advice from the World Bank at a high-level meeting of the Bank Group’s President and the Party Secretary of the Guangdong province at the end of 2007. The report, completed in 2009, was prepared jointly with a team of local experts, financed by the provincial authorities. It generated data and conducted analysis on poverty and inequality that provided new perspectives to the Guangdong authorities on the challenge of shared growth. Particular attention was drawn to the administrative requirements and governance aspects of measures that were suggested. However, the report could have paid more attention to the issue of intergovernmental fiscal relations, as transfers from the provincial government to local administrations and from the central government to the provincial government are essential for the sustainability of social safety net programs. While it may be premature to assess the outcomes of this analytical work, so far the diagnosis and recommendations of the report have had a positive impact and are seen by the Guangdong authorities as valuable policy guidance. Counterparts highlighted that Bank recommendations are being implemented gradually, with focus on priority areas, including strengthening of public services and basic health care services, as suggested in the Bank report. In particular, in line with the recommendations of the report, the poverty line was increased from 1,500 Renmibi to 2,500 Renminbi. In December 2009 the provincial government launched a program of “Equalization of Basic Public Services in Guangdong,” committing 2.5 trillion Renminbi to eight basic public services between 2010 and 2020, with the aim of leveling off differences in service quality and access in urban and rural areas. In health care, 96 percent of the population has been enrolled in the medical insurance scheme while investments were made in the construction of hospitals and health centers and in the training of health staff to better respond to the needs of the insured population. There are indications that inequality in Guangdong has declined in recent years. Between 2008 and 2012, the proportion of urban to rural incomes in the province has declined from 2.96 to 2.67. In urban areas, over the same period, the ratio of the income of the highest and the lowest 20 percent percentiles of the population has declined from 6.8 to 5.3. 48 CHAPTER 4 RESULTS AND THEIR SUSTAINABILITY 4.8 In Chile, the Bank’s knowledge activity on higher education, conducted jointly with the Organisation for Economic Co-operation and Development (OECD), included a nationwide review and a review specific to the Bío Bío region. These have served as a shared reference for all stakeholders for ongoing reform efforts, while the Bank’s work on student loans provided background for recent changes in policies. Another example of evidence-based knowledge services that led to results was the Water Quality Monitoring TA in the Russian Federation: techniques were tested in the Iset River Basin in parallel with preparation of a major revision to the Russian Federation’s Federal Water Law. The knowledge generated from the Iset River Basin was subsequently transferred to assessments and mitigation efforts in 10 other river basins, including the Amur, Don, Koma, Neva, Ob, Ural, and Volga Basins. In Kazakhstan, the Bank’s knowledge activities have contributed to building technical capacity in social protection (by providing, for example, Pension Reform Options Simulation Toolkit training for pensions) and led to policy changes. The introduction of a basic pension and consolidation of pension accounts were implemented as result of Bank recommendations. 4.9 Where the Bank’s advice was well-circumscribed, most often—but not exclusively—in the financial and infrastructure sectors, technical work helped achieve results. The Bank’s Report on the Observance of Standards and Codes (ROSC) on accounting and audit standards in Kazakhstan provided the framework for the implementation of International Financial Reporting Standards and International Accounting Standards, which has been successfully achieved for all large companies and financial institutions. Similarly, in Chile, following up on the Financial Sector Assessment Program (FSAP) recommendations, with support from the Financial Sector Reform and Strengthening (FIRST) initiative, all listed companies have switched to the International Financial Reporting Standards, while progress toward risk-based supervision in the non-banking sector has been significant. In the Russian Federation, the TA on the concession via public-private partnerships of facilities in Pulkovo airport, St. Petersburg, was highly successful, as it led to a contract with a group of investors and facilitated synergy with IFC through an investment for improving and managing those facilities. In Kuwait, although expected results have yet to materialize for most knowledge activities, TA to help the authorities articulate a strategy for emissions abatement was successful in positioning Kuwait as an applicant qualified to obtain Certificates of Emission Reduction under the Clean Development Mechanism of the Kyoto Protocol. In Thailand, the TA on risk-based tax audits has helped the Tax Revenue Department to develop new criteria for taxpayer classification that have already produced results in shortening the value-added tax refund time for big exporters. Similarly, the TA on 49 CHAPTER 4 RESULTS AND THEIR SUSTAINABILITY procurement contributed to a better understanding of the scope of e-auctions and to the process of establishing a centralized procurement unit. 4.10 Results from knowledge activities have been partial in several cases, especially when the thematic area of the Bank’s advice was broad in scope, encompassing an ambitious reform agenda, or when the achievement of results would necessitate multisector efforts. For example, in Bulgaria, results have been partial in education where progress in vocational and higher education is lagging, while in general education achievements have been concentrated in financing and school management, with education quality and student performance still lagging by international comparison. Similarly, while the regulatory framework of labor markets has been rationalized in line with advice by the Bank, skills gaps are prevalent and labor force participation remains weak. In Kazakhstan, the rationalization of higher education institutions was carried out, but not through the establishment of a quality assurance system, which was the key recommendation of the World Bank/OECD Reviews of National Policies for Education report. 22 In Thailand, the Bank’s review of tax policy was used as an input to the government’s 2011 tax reform, although only a limited number of recommendations have been so far implemented—such as, for example, reducing the corporate income tax but with no meaningful revision of holidays from the tax. 4.11 Although lending was not at the core of the partnership in the evaluation’s focus countries, it remained a powerful driver of results for the Bank’s knowledge services. The review reveals that at least partial achievement of expected outcomes of Bank knowledge activities was observed more frequently when activities were used for the design of lending operations. As borrowing in the focus countries is relatively limited, only 39 out of the 196 knowledge activities reviewed were used at least partly in support of the preparation of lending operations. In 95 percent of these knowledge activities, the expected outcomes at inception were likely to be at least partly achieved, against 81 percent in the case of knowledge activities not associated with lending operations (figure 4.2). 50 CHAPTER 4 RESULTS AND THEIR SUSTAINABILITY Figure 4.2. Achievement of Expected Outcomes of World Bank Knowledge Activities (in % of Reviewed Projects) Total World Bank knowledge activities reviewed by IEG AAA financed by the World Bank (including trust funds) AAA cofinanced with the client (including entirely funded by reimbursable services) Outcomes partly achieved Outcomes achieved AAA at least partly used to support lending operations AAA not associated with lending 0 25 50 75 100 % Source: World Bank AAA review. Note: AAA = analytical and advisory activities, IEG = Independent Evaluation Group. 4.12 By contrast, the achievement of outcomes was not correlated with financing arrangements for Bank knowledge services. Among the 196 knowledge activities reviewed, 127 activities were financed by the Bank’s budget (including trust funds) and 69 were cofinanced by the client (including 100 percent RAS). While outcomes were at least partly achieved in about 86 percent of Bank-financed knowledge services, this proportion was slightly lower, at 81 percent, in the case of knowledge services cofinanced or entirely financed by the client (figure 4.2). Reimbursable knowledge services are in principle expected to achieve better results as they are demand-driven and should benefit from stronger client focus and perhaps also stronger Bank effort. The absence of significant correlation between achievement of results and the mode of knowledge service financing reflects perhaps the high relevance to the client of Bank knowledge services in the focus countries with knowledge-based country programs, regardless of the financing arrangements underpinning the Bank’s knowledge work. As noted in chapter 2, the majority of Bank knowledge services in these countries were demand-driven, as 81 percent of knowledge services was requested by the client, while client participation was very high in several stages of knowledge service preparation. When relevance and client participation are high, the mode of knowledge service financing appears to be less critical for the achievement of results. 4.13 IFC projects had mixed results. In China, IFC achieved results by using a programmatic approach to developing a credit information system and secured transactions regime, with the initial project focusing on raising awareness and later 51 CHAPTER 4 RESULTS AND THEIR SUSTAINABILITY projects supporting various stages of development and implementation. IFC also focused its efforts on supporting investment promotion in Sichuan province with several projects, helping the province become a major destination for foreign direct investment. In the Russian Federation, two projects on securitization did not produce expected outcomes as a result of the financial crisis, which delayed enabling legislative and regulatory reform. However, there is high-level interest in the work on the subnational Doing Business database, which policy makers view as a useful tool for measuring regional performance in improving the investment climate. In addition, the three projects on reforming real estate procedures are beginning to show results as regions begin to address administrative barriers to private investment. In South Africa, there were disagreements or lack of interest from counterparts in IFC’s work on competition policies and administrative barriers. However, projects on tax burden and compliance costs to micro, small, and medium enterprises (MSME) resulted in significant changes in tax policies. A project on credit information for microfinance was also successful, and there are plans to scale up the model to other segments of the financial system. Factors Associated with the Achievement of Results of World Bank Knowledge Services 4.14 What factors are behind the achievement of outcomes of Bank knowledge services in countries with knowledge-based country programs? IEG’s review focused on Bank knowledge services as the sample is large enough (196 activities) to permit robust analysis according to the degree of outcome achievement. The sample of IFC Advisory Services reviewed, consisting of only 32 projects, is too limited to allow robust analysis. Bank knowledge activities reviewed by IEG were classified according to the achievement of their expected outcomes (as portrayed in table 4.1): activities with outcomes expected to be fully achieved, activities with outcomes partly achieved, and activities where expected outcomes were not achieved. 4.15 The expectation underpinning the approach taken in this report (see chapter 1) was that relevance (objectives and design) and quality (global expertise, customization, and actionable recommendations) will surface as key factors for the outcomes. Quality and relevance to the client indeed appear to matter for the achievement of expected outcomes (Table 4.2): Activities with outcomes likely to be achieved more frequently had designs appropriate to meet their objectives and were requested by the client compared to knowledge activities with outcomes partly achieved or not achieved. Furthermore, activities with outcomes not achieved more often were unable to address relevant issues to the client compared to knowledge services with outcomes likely to be achieved. Activities lagging in the achievement 52 CHAPTER 4 RESULTS AND THEIR SUSTAINABILITY of outcomes were also weak in conveying international best practice, providing relevant examples, producing new evidence and data useful for policy making, formulating actionable recommendations, and discussing the capacity requirements and administrative feasibility of the implementation of recommendations. Moreover, lack of timely delivery of knowledge services to affect important decisions appears to be associated with the failure to achieve the expected outcomes. There are abundant examples of knowledge services that achieved results and met very high standards of relevance and quality, but there are also counter examples of failure to achieve these standards and that confirm the rule that quality matters for results and highlight practices that need more focused attention by staff and management (see chapter 3). Table 4.2. Degree of Achievement of Knowledge Services Expected Outcomes in Relation to Knowledge Service Characteristics Evaluated by IEG (in % of Knowledge Services in Each Category of Outcome Achievement) Indicators Expected outcome of AAA likely to be achieved Yes Partly No Design of knowledge activity appropriate to meet its objectives 100 88.9 76 Activity requested or commissioned by client 87.1 80.6 72 Activity was unable or partly unable to address some issues relevant 17.3 32 40 to the client Activity conveyed international best practice and relevant examples 87.1 73.8 52 No new evidence or data generated to inform decision making 5.4 10 28 Local expertise used by activity 54.8 50 28 Actionable recommendations consistent with findings were provided 72 73.6 52 Capacity requirements and administrative feasibility of 54.8 68 52 recommendations at least partly discussed Activity outputs not delivered in time to inform important decisions 1.1 6.9 28 Activity disseminated and covered by media 65.6 52.8 36 Activity contributed to developing or strengthening institutions 51.6 31.9 16 Activity contributed to strengthening analytical or policy formulation 71 45.8 20 capacity of recipient Source: Independent Evaluation Group (IEG) knowledge activity reviews. Note: AAA = analytical and advisory activities. Some questions on the tasks (such as, Was the activity requested by the client?) were more objective than others (such as, Was the design appropriate to meet objectives?) and accordingly were less subject to observation errors. The indicators in the table average the responses across all tasks and are likely to average out the observation errors of responses on the individual knowledge activity 4.16 Drawing on local expertise appears to be a significant factor of results as participation of local experts was higher in knowledge services with outcomes fully or partly achieved as opposed to unsuccessful knowledge services. Dissemination of knowledge services was more frequent when outcomes were achieved than when 53 CHAPTER 4 RESULTS AND THEIR SUSTAINABILITY they were not, although the causation may also run in the opposite direction to the extent that achieved results tend to be publicized. IEG’s review also reveals that knowledge services that achieved results have more often contributed to strengthening institutions as well as the analytical and policy formulation capacity of recipients (table 4.2). Stronger institutions and capacity, in turn, would be expected to contribute to the sustainability of results. Figure 4.3. Client Participation in Knowledge Activity Stages in Relation to Degree of Achievement of Knowledge Activity Expected Outcomes (in % of Reviewed Knowledge Activities in Each Category of Outcome Achievement) Scoping Design Implementation Analysis of results Outcomes not achieved Outcomes partly achieved Formulation of recommendations Outcomes achieved Review and feedback Dissemination 0 10 20 30 40 50 60 70 80 90 % 4.17 IEG’s review reveals that client participation in the various stages of knowledge activities was an important factor associated with the achievement of results (table 4.3). In 60 to 80 percent of knowledge services where expected outcomes were achieved, client participation was noted in all stages of the activities. When knowledge service outcomes were partly achieved, client participation was observed less frequently, depending on the stage of the activity. Client participation was even less frequent when results were not achieved. When results were achieved, client participation was comparatively greater in the stages of implementation, analysis of results, and formulation of the recommendations. Building client ownership of knowledge services thus appears to be a key factor of results, together with the participation of local experts. Interlocutors often mentioned that the use of local experts and counterpart participation help ensure that global best practices are 54 CHAPTER 4 RESULTS AND THEIR SUSTAINABILITY modified to take into account local conditions, recommendations take into account capacity constraints, and findings and suggested actions have ownership by stakeholders. In Malaysia, for example, client ownership of knowledge services delivered through fee-based services was strong and a local team was put together, including technical and steering committees, to work with the Bank. However, the knowledge activity, according to some, did not ensure appropriate depth and breadth of country knowledge as the work was still primarily done by international consultants. More engagement of local experts in tasks beyond support functions, such as collecting data, could have improved the relevance of the analysis (by providing, for example, more historical perspective of the phenomenon of brain drain) and the feasibility of the recommendations (for example, in the knowledge economy activity); and could have strengthened knowledge transfers (for example, in the modeling techniques used in the value chain analysis). 4.18 Bank-facilitated brainstorming sessions and seminars with high-level government officials are knowledge activities with strong client participation. They have been organized regularly in Kazakhstan, as a cornerstone of the Joint Economic Research Program, and they have also been organized in Malaysia. Brainstorming sessions in Kazakhstan appear to have been optimized by bringing together panels of high-level experts, providing clear policy messages based on international experience and customized to Kazakhstan. Although no formal framework to track results is in place, the brainstorming sessions appear to have had impact on issues such as resuming preschool education services, elaborating an anti-crisis package, establishing per capita financing in the health system, and providing impetus to an urbanization program. However, while brainstorming sessions are valuable for dialogue and can be highly influential on policies, they risk becoming a confidential exercise where the Bank’s advice may lack impact because of political economy considerations, vested interests, or insufficient engagement of a broader set of stakeholders. 23 Options to mitigate this risk would include organizing open sessions, involving more interested stakeholders depending on the various topics, and keeping minutes of the sessions that could be disclosed, in consultation with the authorities, after a specified time. 4.19 Although the small size of IFC Advisory Services reviewed makes a systematic analysis of factors associated with results less robust, several factors identified in IEG’s reviews were critical to the achievement of outcomes of IFC Advisory Services projects: Client ownership, clear objectives, quality of design, and the quality of monitoring and evaluation (M&E) systems were prominent among these factors. Use of programmatic approaches and strong links with IFC investment projects were also important. Paradoxically, some Advisory Services projects that were launched in response to demand from certain agencies did not produce strong 55 CHAPTER 4 RESULTS AND THEIR SUSTAINABILITY results, sometimes because these were one-off initiatives with poor sustainability prospects. The project on insurance contractual law in the Russian Federation and the support to Endeavour, an entrepreneurship program in South Africa, were responses to client demand, but are not likely to achieve their intended outcomes. Use of projects to raise awareness or support pilot activities was critical to securing ownership and demonstrating benefits. The initial Advisory Services on secured transactions was not a response to a client request, but was able to generate client ownership for the creation of modern regime, supported by subsequent Advisory Services projects. Quality of design was a major success factor in the small and medium enterprises (SME) Banking project in South Africa, which established a microfinance credit information system. The project involved participation of banks where IFC had investments to demonstrate the benefits of using credit bureaus. Inability to restructure Advisory Services projects in the face of changing context adversely affected outcomes, as in the case of the Sichuan Recovery Investment Promotion project where the needs of the client evolved, and the project objectives and outcomes needed to be modified but were not. Sustainability and Monitoring of Results Table 4.3. Sustainability of Results and Capacity Building Outcomes of Bank Group Knowledge Services Percentage Indicators Categories NA No Partly Yes World Bank Knowledge 10.2 9.2 25.5 50 Outcomes of activity likely to be sustained Activities IFC Advisory Services 6.3 15.6 18.8 56.3 World Bank Knowledge 2 38.3 11.2 48 Knowledge activity complemented by other Bank Activities Group activities including lending IFC Advisory Services 3.1 21.9 12.5 62.5 World Bank Knowledge 10.7 27 22.4 38.3 Activity contributed to developing or strengthening Activities institutions IFC Advisory Services 21.9 46.9 9.4 21.9 World Bank Knowledge 4.1 17.3 25 53.1 Activity contributed to strengthening analytical or policy Activities formulation capacity of recipients IFC Advisory Services 25 15.6 18.8 40.6 World Bank Knowledge 39.8 39.3 13.3 7.1 Achievement of agreed capacity development Activities outcomes monitored, and lessons learned used IFC Advisory Services 59.4 12.5 15.6 12.5 Source: IEG knowledge activity reviews. Note: NA= Not Applicable; Some questions on the tasks (such as, Was the activity requested by the client?) were more objective than others (such as, Was the design appropriate to meet objectives?) and accordingly were less subject to observation errors. The indicators in the table average the responses across all tasks and are likely to average out the observation errors of responses on the individual knowledge activity. 56 CHAPTER 4 RESULTS AND THEIR SUSTAINABILITY 4.20 Sustainability of results of Bank Group knowledge services was in line with their achievement of results. The outcomes of 75 percent of Bank knowledge services and IFC Advisory Services reviewed by IEG were likely to be at least partly sustained (table 4.3). A majority of Bank knowledge services and IFC Advisory Services were complemented by other Bank Group activities, which may have contributed to the sustainability of outcomes. IEG’s review reveals that 60 percent of Bank knowledge services contributed at least partly to developing or strengthening institutions. Similarly, a large majority of Bank knowledge services and a significant part of IFC Advisory Services contributed to strengthening analytical or policy formulation capacity of recipients. Institution building and capacity development raise the likelihood of sustainable results. Monitoring of capacity development objectives was, however, weak, especially for Bank knowledge services and less so for IFC Advisory Services. Capacity development outcomes in Bank knowledge services, when applicable, were at least partly monitored only in one-third of knowledge services reviewed by IEG. Furthermore, they were at least partly monitored in more than two-thirds of IFC Advisory Services reviewed. Table 4.4. Sustainability of Knowledge Activity Outcomes in Relation to Knowledge Activity Characteristics Evaluated by IEG (in % of Knowledge Activity by Degree of Outcome Sustainability) Outcome of knowledge activity likely to be Indicators sustained Yes Partly No Activity conveyed international best practice and relevant examples 84.7 72 66.7 New evidence or data generated to inform decision making 78.6 64 50 Local expertise used by activity 60.2 40 27.8 Actionable recommendations consistent with findings were provided 74.5 74 55.6 Capacity requirements and administrative feasibility of 50 46 22.2 recommendations discussed Knowledge activity complemented by other World Bank activities 60.2 38 22.2 Activity contributed to developing or strengthening institutions 56.1 30 5.6 Activity contributed to strengthening analytical or policy formulation 68.4 52 5.6 capacity of recipient Source: Independent Evaluation Group (IEG) knowledge activity reviews. Note: Some questions on the tasks (such as, Was the activity requested by the client?) were more objective than others (such as, Was the design appropriate to meet objectives?) and accordingly were less subject to observation errors. The indicators in the table average the responses across all tasks and are likely to average out the observation errors of responses on the individual knowledge activity. 4.21 The factors analyzed above, associated with the achievement of knowledge service outcomes, were also correlated with the sustainability of the outcomes (table 4.4). In the majority of cases where outcomes were likely to be sustained over time the knowledge activity conveyed international best practice and relevant examples, generated new evidence to inform policy making, and formulated actionable 57 CHAPTER 4 RESULTS AND THEIR SUSTAINABILITY recommendations consistent with the findings. This was less often observed when knowledge activity results were partly likely or unlikely to be sustained. Similarly, when the results were likely to be sustained, knowledge services more often used local experts and discussed the capacity requirements and administrative feasibility of implementing the recommendations. Counterparts often said that use of local experts enhances the sustainability of knowledge activity results as it improves the applicability of the recommendations and helps build the local capacity needed for long-term impact. In addition, sustainability of outcomes was more often observed when knowledge services were complemented by other World Bank activities (lending, other ESW, or complementary TA). In the majority of cases where sustainability of outcomes was likely, knowledge services contributed to strengthening institutions or the analytical and policy formulation capacity of recipients. This was observed less often when outcomes were partly sustainable and was virtually not observed when sustainability was unlikely (box 4.3). Box 4.3. Sustainable Knowledge Activity Outcomes Through Capacity Development and Institution Strengthening: Examples from South Africa and China The South Africa Micro, Small, and Medium Enterprises (MSME) Study, conducted in 2004- 06, was a follow-up to the Investment Climate Assessment. It reviewed the effectiveness of existing programs supporting MSME; provided new information through surveys; introduced new methodologies, such as the value chain approach; and provided actionable recommendations. While World Bank staff led the study, several local consultants prepared parts of the study, including the surveys and value-added analysis. The study was a major input to the government’s MSME strategy, including an evaluation of the design of the MSME program. The Small Enterprise Development Agency was subsequently created to implement the MSME strategy, with the Department of Trade and Industry refocusing on the policy-making and strategy formulation functions. These changes strengthened the institutional framework for MSME development. At the same time, the study provided new methodologies (value chain analysis, evaluation of programs) that would be useful in policy formulation. In a very different context, at the request of the authorities, the Bank provided support to the International Poverty Reduction Center in China (IPRCC) through three experience- sharing events from 2008 to 2010. These events aimed to facilitate the sharing of China’s development experience with African policy makers and development practitioners. The activity led to a set of case studies and briefing notes that were used in the events and published by the client as training materials. Most presenters in these events were Chinese government officials, researchers from Chinese institutes, professors from Chinese universities, or IPRCC staff. The outputs were used to build the capacity of IPRCC and raise African participants’ awareness of China’s development experience. Another Bank TA activity, to the China-Development Assistance Committee (DAC) Study Group, worked with IPRCC on related topics. These two activities reinforced each other as the objective of the China-DAC Study Group was to help participants reach consensus on what is important in development assistance. IPRCC’s analytical capacity improved through this training and 58 CHAPTER 4 RESULTS AND THEIR SUSTAINABILITY research platform and this improvement is expected to be sustained. Both Bank activities contributed to building IPRCC’s capacity to design and deliver training programs in the future. At the same time, knowledge gained by African officials is likely to benefit them in the long run. Although there were no clear indicators to measure the outcomes of the events, this activity, and its companion TA to the China-DAC Study Group, are among the few reviewed by the Independent evaluation Group (IEG) to have promoted South-South policy exchanges. Source: IEG knowledge activity reviews. 4.22 Bank knowledge service results were not monitored and evaluated consistently for either the individual activity or the CPS. 24 IEG’s review indicates that just 17 percent of knowledge services assessed in the focus countries had at least a partial results framework in the CPS making it possible to track the contribution of the activity to the broader development outcomes sought by the CPS (table 4.5). Similarly, only 23 percent of knowledge services included, at least partly, result indicators to track the achievement of the activity’s outcomes. It is notable that, in contrast to Bank knowledge services, the majority of IFC Advisory Services reviewed by IEG were at least partly equipped with results indicators to trace achievement of outcomes. Although monitoring of capacity development outcomes and lessons learning were weak for Bank knowledge services, it appears that they were less so for knowledge services with outcomes likely to be achieved. Table 4.5. Results Indicators of World Bank Group Knowledge Services Indicators Categories Percentage NA No Partly Yes World Bank 7.1 75.5 5.6 11.2 Results indicators in CPS to measure the knowledge Knowledge Activities activity contribution to CPS development outcomes IFC Advisory 3.1 37.5 9.4 50 Services World Bank 11.7 64.8 10.2 12.8 Measureable results indicators included in Knowledge Activities knowledge activities IFC Advisory 6.3 12.5 18.8 62.5 Services Source: Independent Evaluation Group knowledge activity reviews. Note: NA = Not Applicable. Some questions on the tasks (such as, Was the activity requested by the client?) were more objective than others (such as, Was the design appropriate to meet objectives?) and accordingly were less subject to observation errors. The indicators in the table average the responses across all tasks and are likely to average out the observation errors of responses on the individual knowledge activity 59 CHAPTER 4 RESULTS AND THEIR SUSTAINABILITY Synergy Between World Bank and IFC Knowledge Services in Achieving Results 4.23 World Bank knowledge services and IFC Advisory Services 25 generally complement each other in contributing to results. Several factors influence the degree to which World Bank and IFC knowledge services have synergy. The main factor is the quality of the CPS results framework, in particular the articulation of CPS outcomes and the clarity of the linkage between the Bank Group’s programs, projects, and instruments with the CPS outcomes. Another factor is the existence of core ESW—such as ICAs and FSAPs—to underpin the strategy and help identify priorities for improving the investment climate and developing the financial sector. In a few cases, there were well-defined programs of joint World Bank and IFC knowledge activities. Nonetheless, the experience with coordination between the World Bank and IFC was mixed. There is scope for improving coordination between the World Bank and IFC, for example, by establishing more systematic mechanisms for inclusion of one institution in the other’s review processes—especially at the concept and design stages. 4.24 In the case of South Africa, for example, all World Bank and IFC knowledge services covered by the review were intended to contribute to the private sector development component of the first pillar of the CPS, with the outcome defined as improved investment climate for labor-intensive informal and formal enterprises, focusing on MSME. Two ICAs and a major MSME ESW by the Bank were the core analytic work, followed by a series of World Bank and IFC knowledge activities that enhanced knowledge in various issues or themes identified in the core ESW. The distribution of follow-up work was based on the respective institutions’ areas of expertise and experience, with the World Bank focusing on deeper analysis of exchange rate and labor regulations and IFC dealing with issues relating to administrative barriers or compliance costs faced by small businesses. In addition, IFC provided TA to three private sector institutions supporting small businesses, including two entrepreneurship programs, building on IFC experience with similar projects in the region. There are other thematic areas where the World Bank typically takes the lead, such as on tax issues, but IFC conducted two projects on MSME tax burden and compliance. To ensure synergy and coordination, IFC used experienced Bank staff to manage the projects. However, there seems to be a lack of clarity and poor coordination in the support to the Department of Trade and Industry where both World Bank and IFC provided TA—the World Bank, for example, had not received the report IFC prepared on competition policy. 4.25 In the South Africa financial sector, only one IFC project was reviewed, a three-year TA program to support MSME lending by improving credit information for smaller firms, an area where IFC has experience. Establishing credit bureaus and 60 CHAPTER 4 RESULTS AND THEIR SUSTAINABILITY secured transactions regimes have become a major focus of the access to finance business line in Advisory Services. In addition, IFC is able to use its investments in the financial sector in South Africa to design a project that supports both the supply and demand side of credit information. In the process, it provides a demonstration of the benefits to encourage wider use of such information. The IFC Advisory Services in the financial sector support the CPS strategic component and the Bank’s ESW on MSME. There is no overlap with other World Bank financial sector knowledge activity, which included an FSAP report and an ROSC on insolvency. Nonetheless, interviews with Bank staff indicated that coordination could be improved, especially given the increasing focus by the Bank on MSME financing. 4.26 In the case of the Russian Federation, the CPS aimed at strong progress in land registration and cadaster listings as a primary outcome of the pillar on sustaining rapid growth. Three of the five IFC activities reviewed focused on advancing administrative reforms in land or real estate transactions and complemented the World Bank Cadaster Project. In addition, two of the IFC real estate projects were managed by Bank staff, which reduced the risk of lack of coordination between the two institutions. Linked to the work on land is the IFC project to help develop the primary mortgage market, which would enable growth of secondary mortgage market and in the process deepen and broaden financial markets. As noted in the CPS Completion Report Review by IEG, the International Property Rights Index has improved slightly since the simplification of procedures and creation of the legal basis for a Unified Real Estate Cadaster. In addition, the World Bank’s ICA provided an important basis for selecting Bank Group activities. World Bank knowledge activities reviewed by IEG were mainly in economic diversification, competition, and innovation, where the World Bank typically takes the lead, with IFC focusing on administrative barriers to investments, including an IFC project on lessons learned in reforming administrative barriers in the Russian Federation aimed to elicit discussion on effective strategies. In response to the government’s demand for regional analysis in the investment climate work, the World Bank and IFC jointly delivered the Subnational Doing Business report, which was well received. 4.27 There was less coherence and coordination in the World Bank and IFC financial sector work in the Russian Federation in support of the CPS expected outcome of increased access to finance for entrepreneurs. The IFC work on securitization and insurance was not aligned with CPS emphasis on MSME financing, which was a major focus of IFC investments and supported by a World Bank study. The work on securitization did not produce the outcomes envisioned due to the financial crisis. In the case of insurance, the consolidation of the regulatory agencies left the ongoing IFC work uncertain on the way forward. One 61 CHAPTER 4 RESULTS AND THEIR SUSTAINABILITY exception noted above is the work on mortgage development, which was linked to the land reforms and IFC investments in regional banks with mortgage lending portfolios. The World Bank AAA in the financial sector revolved around issues flagged in the FSAP Update, and was accompanied by work on crisis response, consumer protection, banking vulnerability assessment, payment system, and macro prudential tools. The Country Partnerships Strategy Completion report (CPSCR) Review by IEG concluded that the increased flow of financing to Russian entrepreneurs did not materialize as expected, though there were some improvements in capital market regulations. 4.28 In China, the World Bank ICA identified priorities to improve the investment climate in 120 cities. Five IFC projects complemented the ICA in two ways. First, an IFC project provided a monitoring tool for tracking the performance of various regions and cities, including the establishment of a baseline. Second, IFC provided TA—including to Sichuan province—to help identify and implement best practices in investment promotion. The work with Sichuan province, which has been experiencing significant increases in foreign direct investment flows over the past five years with IFC Advisory Services and investment support, would provide a demonstration of various investment promotion practices that could be replicated, with the help of central authorities, in other provinces. Nonetheless, there was only modest progress overall in simplifying cumbersome business regulations. 26 There was also an IFC project that supported the government in shaping its policies on corporate behavior, specifically in the areas of corporate governance, labor, and environmental and social practices. The Advisory Services contributed to the CPS outcome on adoption of best practice in corporate governance and social responsibility by publicly listed state-owned enterprises and large private corporations. Guidelines on corporate behavior have been issued for listed firms as well as state-owned enterprises under the State-Owned Assets Supervision and Administration Commission. 4.29 In the financial sector, World Bank knowledge services focused on the enabling environment for capital markets development while IFC Advisory Services projects were mainly addressing access to financial services, in line with the priorities in the CPS pillar on financing sustained and efficient growth. The World Bank produced the FSAP as well as knowledge products on financial stability, creditor rights, accounting and auditing, and capital markets. The World Bank also produced knowledge products on MSME financing—specifically, two papers for the People’s Bank of China on microfinance and lending through intermediaries, to pave the way for a Bank loan to China Development Bank for MSME. IFC followed a programmatic approach to its work on MSME and complemented the Bank’s knowledge and lending activities. The IFC project on SME Access to Finance was a 62 CHAPTER 4 RESULTS AND THEIR SUSTAINABILITY focused nationwide survey to inform policy dialogue and raise stakeholder awareness. Three projects supported the development of a credit information system and secured transactions regime, starting with a review of the current institutional framework, followed by establishment of a pilot registry and finally by support for the operational phase of the system. As of the end of 2011, the credit reporting system covered more than 600 million individuals and another 18 million entities while the secured transactions system contributed to the 24 percent annual increase in loans using moveable assets during 2008-2010. Other IFC Advisory Services in the financial sector provided support for leasing and housing finance to improve access to financial services by introducing new instruments or lending products. Finally, the IFC Advisory Services leading to the enactment of the Personal Bankruptcy Law would help strengthen the legal framework affecting MSME lending. Interviews with staff indicate satisfactory coordination between IFC and World Bank Finance and Private Sector Development staff in Beijing. 63 5. Conclusions 5.1 The Independent Evaluation Group’s (IEG’s) review of knowledge-based World Bank country programs has aimed to discern the factors contributing to knowledge service effectiveness. It finds that Bank Group knowledge activities have been strategically relevant in countries that do not rely much on its financial services. Clients value the Bank Group’s ability to convey international best practice, act as a trusted knowledge broker, customize knowledge to the local context, and take a sensible approach to important multisectoral development issues. At the same time, the Bank Group’s knowledge services in the focus countries, delivered through the Bank’s knowledge services and the International Finance Corporation’s (IFC’s) Advisory Services, have a reasonably good record of achieving results with a fair likelihood of sustainability. As more emerging market economies are graduating from the World Bank’s financial services and demanding programs with higher knowledge content, the Bank Group’s challenge will be to build on this record to ensure that it remains a partner of choice for upper-middle-income countries. 5.2 To respond to this multifaceted challenge, the World Bank should organize internally to meet effectively the growing demand for knowledge-based partnerships, while offering a set of customized services through delivery mechanisms that help clients achieve sustainable results. This will involve well- informed choices in the way the Bank responds to multiple client demands, allocates limited own human and financial resources to knowledge-based partnerships, expands the frontier of engagement through appropriate use of Reimbursable Advisory Services (RAS), and works with clients to track the achievement of results. Leveraging the knowledge acquired from these partnerships to make them available to lower-income countries would be an additional challenge given the vision the Bank Group aspires to fulfill as a Bank for the whole world. 5.3 This chapter summarizes the main lessons learned from the evaluation as they relate to three questions: • How can the World Bank enhance its value proposition in knowledge-based partnerships? • How can the World Bank delineate the scope and instruments of knowledge- based country programs? • How can the Bank Group leverage its engagement in knowledge-based country programs? 64 CHAPTER 5 CONCLUSIONS How Can the World Bank Enhance Its Value Proposition in Knowledge-Based Partnerships? 5.4 Three preconditions seem particularly important for the World Bank to add more value for clients seeking a partnership with higher knowledge services content: • Paying enough attention to the quality and relevance of knowledge services delivered to clients • Emphasizing practical, “how to” advice in the form of options for action • Understanding incentives and the political economy of reform and making good use of local expertise. 5.5 Attention to the quality and relevance of knowledge services is key for results—regardless of how those services are financed—and for shaping learning under the “science of delivery” that is central to the present Bank reform effort. Focus on quality is important for results in all Bank Group country partnerships, but it is paramount when results depend on knowledge services alone, with no support from financial resources or procurement of goods and services as in Bank lending operations. IEG’s review revealed that the quality and relevance of Bank knowledge services appear to be correlated with the degree of achievement of expected outcomes, independently of whether knowledge services are funded through the Bank’s own resources, cofinanced by the client, or entirely financed through RAS. Ensuring top-quality knowledge services requires effort on multiple fronts: relevance of design to respond to client concerns, customization of international best practice to local conditions and institutional or capacity constraints, generation of data to support evidence-based policy making, formulation of actionable recommendations that fit local administrative and political economy constraints, and timely product delivery to influence key decisions. The Bank’s knowledge services will need to meet the highest standards on all of these fronts to help clients achieve development results. 5.6 Successful delivery of high-quality knowledge services that meets the above criteria requires highly experienced staff with global perspective and ability to deliver products on time. The quality and the technical competency of the team leader, who has to be an expert in the field, are of critical importance. In addition, the sustainability of RAS greatly depends on the quality products delivered to ensure continued commitment and interest of high-level officials. Therefore, the Bank needs to pay close attention to the expertise and availability of staff to ensure success of the knowledge-based program to generate “value for the government’s money.” 65 CHAPTER 5 CONCLUSIONS 5.7 This was evident in Kuwait, where the level of the team’s expertise proved crucial in the success of knowledge products. Where the Bank team was not well staffed there was reputation damage, as in the bank governance knowledge activity. This was also observed in South Africa, where strong and broad-based domestic consulting capacities are available and where some Bank work was underappreciated or perceived not to be of consistently high quality. The Bank should be cautious in engaging in RAS when the appropriate level of expertise cannot be mobilized in a timely fashion. It is important that the Bank be able to assemble a team that is available to work on the project for its duration, to minimize unnecessary delays due to staff rotation and work effectively with counterpart steering groups. The Bank needs to strengthen the quality assurance process for knowledge services—particularly through a reinvigoration of the internal peer review process—to ensure adequate technical quality of the products delivered to clients. The latter is essential for the Bank to maintain its unique position on the knowledge front through RAS. 27 The Bank should also be mindful of staff incentives, as participation in lending operations is typically seen as more rewarding for career development than participation in teams for the delivery of—even high profile—knowledge services. This may discourage the ablest and most experienced staff from participating in knowledge-based country programs and should be managed as part of staff career development. 5.8 Emphasizing the “how to” options, as opposed to the diagnostics and the “what to do” recommendations. Where knowledge services are delivered through RAS, clients expect focused and practical advice, with short documents that are operational rather than long documents heavy on diagnostics and generic advice, which are useful for the policy-setting agenda but less operational. Presenting options and scenarios of different actions, rather than what is often perceived by counterparts as “recipes,” would enhance the credibility and impact of the Bank’s work in upper-middle-income countries that demand programs with high knowledge content. Providing advice in the form of options for action would enhance client ability to own the final policy decisions, action plan, or strategy. In South Africa, for example, this was true for recommendations in sensitive areas such as labor, MSME policies, and land reform where follow-up discussion of options highlighted by knowledge services (such as ICAs) led to positive results. It is important that the options presented be based on best international practice and a deep understanding of countries’ institutions. Only in this way can policy advice really become operational for counterparts. 5.9 More generally, emphasizing “how to” options may require a different approach in delivering knowledge services, one that involves much more time interacting with clients, access to experienced consultants, and consequently more 66 CHAPTER 5 CONCLUSIONS resources. The experience with the distribution of projects between the World Bank and IFC, with the Bank focusing on policy and IFC on “how to” aspects, may be a workable model in the investment climate and financial sector work. In any case, the Bank may have to review the skills mix of knowledge service teams for meeting the expectations of clients that demand higher knowledge content. This would require a change in attitude by team leaders and consultants, who should be hands-on, practical, and policy-oriented. Teams may need to draw more on Development Economics Vice Presidency (DEC) resources to ensure high-quality research, and on high-level consultant expertise with practical experience to provide practical know- how. Where the Bank has a Country Office, its presence would have to be maintained—although probably resized to the level of client demand—to preserve the Bank’s perceived comparative advantage in customizing international best practice to the local context. With very limited lending in most countries with knowledge-based programs, the sustainability of this business model would require a decisive move toward RAS. 5.10 Making an effort to understand the political economy of reform and maximize the use of local expertise. As in all countries around the world, of particular importance in countries with knowledge-based programs are the incentives of various stakeholders to carry out a reform program. Understanding these incentives could enhance the impact of the Bank’s knowledge services by presenting options that address the concerns and constraints faced by different constituencies. This could, for example, have enhanced the effectiveness of Bank knowledge activity on regional development in Chile. In Kuwait, for some activities, although the Bank’s recommendations were theoretically sound, they were considered not implementable in the current political economy of the country. In complex political environments, as is often the case in countries with knowledge- based World Bank programs, it is important for the Bank to identify and work with champions, provide them with ammunition to stay engaged in internal policy debates, and promote the reform agenda over time. This proved true in South Africa’s private sector where successful projects were led by appropriate partners (small and medium enterprises [SME] banking and pro-poor tourism). 5.11 Bridging the gap between international good practices and local conditions could be facilitated by involving local experts more extensively. This would not only enhance the applicability of the recommendations but also build the local capacity to achieve longer-term impact. Recourse to local expertise would also help the Bank understand better the political economy of reform in specific areas where advice is requested. As revealed by IEG’s review, use of local expertise was associated with a greater degree of achievement of knowledge service results in the focus countries as well as with a greater likelihood of their sustainability. In China, for example, the 67 CHAPTER 5 CONCLUSIONS most effective knowledge products start with a request from the highest level of government. They also involve close collaboration with local experts from the beginning of the project, drawing on their expertise in data collection, analysis and formulation of recommendations, and follow-up. For IFC, programs that start with a project that reviews local context, including use of local working groups, have been successful in generating credibility and ownership of recommendations. How Can the World Bank Delineate the Scope and Instruments of Knowledge- Based Country Programs? 5.12 The scope and content of a knowledge-based country program are typically circumscribed with the aim of matching as best as possible evolving government priorities over the country partnership strategy (CPS) cycle, according to the Bank’s comparative advantage and expertise in responding to client demands. Beyond the guiding principles for the design of the CPS, some key considerations for deciding the appropriate scope of a knowledge-based country program and its instruments would include: • Remaining engaged in specific areas over an appropriate period • Remaining engaged in areas relevant to the client country’s medium-term development agenda, beyond the scope of RAS • Conducting broad consultation with stakeholders and dissemination of knowledge services • Making the most of RAS on the basis of transparent but flexible criteria, to sustain a business model conducive to results • Using implementation and results monitoring systems to track progress toward mutually agreed outcomes. 5.13 Staying engaged and responsive in the implementation phases of Advisory Services through appropriate instruments. IEG’s review reveals that a main factor that often limits the impact of freestanding knowledge services is the lack of a follow-up instrument to support transformation of recommendations into actual policies and programs and support implementation. In the absence of lending, follow-up support should help clients translate recommendations from sound analysis into implementable actions that take into account local political and administrative constraints. Furthermore, in a complex political environment, such as found in most upper-middle-income countries, implementation needs time. To address these issues, IFC has used programmatic approaches that enable a program to be developed within a period commensurate with capacity. Ownership of the technical assistance (TA) program and the sense of urgency for reforms can also be 68 CHAPTER 5 CONCLUSIONS lost with changes in political leadership and the economic environment. The need for “staying the course” was noted, for example, in Kuwait where, although clear results on knowledge services completed over the past five years are elusive, it is encouraging that gradual appropriation of these reforms has been built over time and some key legislation may be closer to being passed. This was facilitated when the Bank team stayed engaged over time to provide advice on best practice legislation—such as in procurement. 5.14 It is possible to design programmatic knowledge services that enable TA or economic and social work (ESW) to follow up on initial work. The work on innovation in the Russian Federation is a good example of such an approach. Initial work provided a set of innovation models to choose from, and follow-up knowledge activities would support design of the specific model chosen. Another example is the Doing Business report in the Russian Federation, where the country Doing Business analysis was followed up by similar regional analysis in response to feedback from stakeholders. Programmatic approaches would be pertinent in work on administrative barriers, economic diversification, small and medium enterprises (SME) development, and public finance management reforms. In Kazakhstan, the recent introduction of a programmatic Joint Economic Research Program (JERP) should help increase the strategic focus of knowledge services and provide room for more sustained sector policy dialogue. Moreover, the move toward a programmatic JERP should help attract and manage experienced Bank staff who will be engaged in a more consistent and challenging work program instead of smaller one-off activities. 5.15 In country partnerships based on knowledge services, engagement during implementation could be sought with a broader range of stakeholders, beyond the established government counterparts. Engaging systematically with civil society organizations (CSOs) could make the World Bank’s knowledge services more effective. The objectives here could be to build CSO capacity, help disseminate the reform agenda, and maintain focus on key policy issues in the public domain. In Bulgaria, a good example of such partnerships was the Bank’s association with the Open Society Institute—a local nongovernmental organization—on the judicial reform agenda, which was very helpful in carrying out relevant field work and disseminating results to the public. The Open Society Institute also has kept the topic in the public domain, which would have been difficult for the Bank to achieve on its own. 5.16 Remaining engaged in areas that are relevant to countries’ medium-term development agendas. One pitfall of the “consultant firm model” that is often being used to respond to client demand through RAS is the possible lack of continuous 69 CHAPTER 5 CONCLUSIONS engagement in some areas as a result of the drive to respond to multiple client demands. Another pitfall is the lack of coverage of thematic areas that may not rank sufficiently high on the short-term priorities of clients so as to be included in jointly financed RAS programs. These drawbacks may risk eroding the Bank’s capacity to see the big picture and provide multisectoral development solutions—a strength of the Bank’s knowledge services that is generally valued by clients. In Chile, for example, stakeholders noted that possible focus areas beyond the Joint Studies Program could include labor productivity, innovation policy, and climate change. The program in Chile is demand-driven, and the client has not requested Bank assistance in these areas, but the broader engagement may be necessary for the Bank to remain relevant in supporting Chile’s development agenda. In Bulgaria, in parallel with helping the government implement the Europe 2020 strategy and absorb European Union (EU) funds, there is a role for the World Bank in continuing to raise awareness on issues beyond the RAS agreement, issues that are important for Bulgaria’s medium-term development agenda, such as judicial sector reform. The Bank will need to think about the appropriate instruments to use (high-level brainstorming, conferences, ESW) and commit necessary resources, incremental to the resources allocated to programs jointly financed with clients. Using the instrument of an annual or bi-annual Economic Monitoring Report with focus on special topics may be an option to consider, in view of the successful record of these publications in Malaysia, Thailand, and China. For IFC, the CPS is an important guide to areas of engagement, making quality of the CPS and its results framework a critical factor in relevance and effectiveness of Advisory Services. 5.17 Conducting a broad consultative process and dissemination, acknowledging the public good function of Bank knowledge services, while paying attention to local circumstances. Based on IEG’s review, client participation in the different stages of knowledge services appears to be closely associated with success in achieving expected knowledge service outcomes. Thus, the Bank should aim at bringing broader participation from various stakeholders into knowledge- based country programs—including by opening up discussions or establishing focus groups with local experts and CSOs. Broad dissemination of knowledge services that have the attributes of a “public knowledge good” (as defined in chapter 2 and Box 2.1), regardless of its mode of financing, would help strengthen the demand side for moving forward on the long-term reform agenda of country clients. Often, due to language and other difficulties, more time is needed for full outreach of the Bank’s knowledge services. A single, short dissemination event may thus not have significant impact. Follow-up thematic discussion sessions may be appropriate and should not be neglected once the Bank’s knowledge product has been formally delivered to the client. 70 CHAPTER 5 CONCLUSIONS 5.18 Broad consultation and dissemination might be even more important in a complex political environment. In Kuwait, for example, presentations by the Bank of diagnostic work and reform recommendations or draft legislation (for example, on procurement and civil service) to government officials and parliament were helpful to build awareness of key issues. In Chile, engaging with a broader range of stakeholders through more informal channels, beyond the direct counterparts involved in the financing and vetting of knowledge services, such as high-level think tanks that exert an influence on the development agenda, would help enhance the impact and sustainability of the Bank’s work. Consultations will need to extend beyond country clients when other development partners are involved. In Bulgaria, for example, as the implementation of the Europe 2020 agenda and the facilitation of the absorption of EU funds are main drivers of the Bank’s new partnership strategy, it will be important to harmonize as much as possible the Bank’s advice with that of EU institutions that frame Bulgaria’s reform agenda. This will require very frequent exchanges with EU institutions at all levels of the Bank’s country team. 5.19 Brainstorming sessions, while highly influential over policies, have the potential to become a confidential exercise where the Bank may be seen as a select advisor to very high-level officials, possibly with vested interests. Ways to avoid this could be to discuss with the authorities the possibility of opening sessions to interested stakeholders as needed, depending on the topic, and keeping full records, which could be disclosed at a specified time. 5.20 Although public demand for Bank analysis is generally high, dissemination is often hampered where knowledge services are provided through RAS. This is because RAS reports are often proprietary and subject to confidentiality. In several countries, such as Malaysia, the Bank has worked hard and with success to persuade the client to release these reports, but many studies remain inaccessible for most stakeholders for at least some time after their completion. The Bank and its clients could explore ways to make Bank studies more widely accessible to increase their impact. One option would be to recognize upfront the public good component inherent in several knowledge products. A portion of the knowledge service cost could be shared by the Bank on the condition that the resulting study would be released to the public shortly after completion. This approach has been considered by the World Bank in the recent review of Fee-Based Services. 5.21 However, it has to be acknowledged that, although broad consultations and strong dissemination efforts are key, the Bank should also be ready to “work behind the scenes” if local circumstances warrant. In South Africa, for example, in some sectors, the Bank should be willing to provide high-quality technical advice to the client on a confidential basis as needed, and be less preoccupied with dissemination. 71 CHAPTER 5 CONCLUSIONS In such cases, the Bank would need to adapt its incentives and policies for recognition of staff work. 5.22 Although RAS do not seem to enable knowledge activities to get more traction on results, they should be systematically offered to clients with a view to ensuring the sustainability of the Bank’s business model in knowledge-based country programs. The World Bank should continue offering RAS to countries that make limited use of its financing services but are seeking its knowledge to an extent that surpasses the Bank’s financial capability to respond to this demand. 28 As the Bank bundles knowledge services with finance in all countries that use its lending services—with the cost of the knowledge services partly reflected in the pricing of the Bank’s loans—offering knowledge services at no cost to countries that make limited use of the Bank’s financing would risk diverting the Bank’s resources away from clients most in need of assistance. Offering RAS is a win-win strategy for the Bank and its clients as it allows the Bank to expand the feasible set of services it provides to countries, while the continuous engagement with upper-middle-income countries on the knowledge front generates new knowledge that the Bank is in a unique position to intermediate to lower-income countries. However, IEG’s review suggests that, although the relevance of RAS is strengthened by client demand and financial commitment, the results achieved do not appear significantly different from those of knowledge services funded by the Bank’s own resources. Other fundamental factors—related to the relevance of design, quality, timeliness, client participation, and use of local expertise—are more closely associated with the achievement of results. 5.23 The Bank’s policy regarding the pricing of RAS stipulates full cost recovery— including direct, indirect, and sustaining costs of the services (overhead). 29 Full cost recovery pricing does not preclude the Bank’s participation in RAS financing. This occurred in practice, to varying degrees, in the evaluation’s focus countries, for example, in Chile and Kazakhstan where the Bank cofinances joint studies programs. The 2012 review of Fee-Based Services conducted by the Bank provides an indicative framework for RAS pricing with the aim of increasing transparency and efficiency. According to this framework, the main criteria for pricing would be the recipient’s income and ability to pay, the recipient’s level of borrowing, public good externalities from disclosure, economies of scale related to the size of RAS, and the cost implications of risk management. 5.24 The new pricing framework is a step in the right direction. It could be complemented by clarifying the types of knowledge services that come close to pure public goods in nature (such as the Economic Monitor series in Malaysia and Thailand, or the Financial Sector Assessment Program [FSAP] and Report on the 72 CHAPTER 5 CONCLUSIONS Observance of Standards and Codes [ROSC] reports), as opposed to those that serve specific needs of counterparts. The Bank should continue to fully fund studies that are pure public goods, while the cost of those studies that mostly benefit the client should be priced at full cost recovery. In countries with knowledge-based country programs, RAS could be offered to institutions that are in position to cover the full cost of the Bank’s services. In Thailand, for example, this could be the case for self- funded organizations such as the Bank of Thailand (work on financial inclusion), the Credit Guarantee Corporation (work on SME master plan), and the Bank for Agriculture and Agricultural Cooperatives (work on risk management). Cost sharing of knowledge services that are not pure public goods could be determined based on the criteria selected by the Bank, with emphasis on the relevance of the activities for other countries and their disclosure by the recipient. The income criterion would also be important, especially to equalize access to the Bank’s knowledge services among subnational clients with varying capacity to pay for the service—as in the case of the Russian Federation’s regions. In addition, full cost recovery could include the cost of a client survey, to be conducted when the knowledge activity is completed with the aim of informing both the Bank and the client about the quality and relevance of the product, its use, and the results achieved or likely to be achieved. This would help the Bank to better track the quality of RAS supplied and learn more systematically about their use and the results achieved so as to better tailor them to client expectations over time. 5.25 Using implementation and results monitoring systems to track progress toward mutually agreed outcomes and mitigate the risk of fragmentation and loss of strategic focus intrinsic in RAS. RAS require flexibility to respond to changing client demand. But, on the downside, fragmentation of RAS programs often results from the commendable drive to accommodate multiple and unforeseen client needs. Flexibility would thus need to be balanced against the risk of fragmentation and an ensuing loss of strategic focus of the Bank’s knowledge work through RAS programs. In the Russian Federation, for example, the considerable increase in knowledge services conducted under RAS since FY08 has reduced the predictability of the knowledge service program. Several of the originally planned knowledge activities in the FY07-11 CPS were dropped and new tasks were added. This could dilute the focus on the achievement of results sought by the country partnership. To mitigate these risks, first, it would be useful to elaborate a CPS in countries where knowledge-based country programs are being implemented without a CPS framework. A CPS with these countries does not have to be the outcome of a burdensome consultation process conducive to an elaborate document. The purpose of the CPS could be to provide guidance on the strategic priorities for World Bank Group engagement in the country to best leverage the expertise of the World Bank 73 CHAPTER 5 CONCLUSIONS and IFC, avoid fragmentation of activities, and help plan efficiently the use of limited Bank resources. It would be, however, important to complement the CPS by a solid results framework. It is sometimes argued that sustained client demand for RAS is the best indication that a knowledge-based country program remains relevant and thus a CPS with a formal results framework could be redundant. However, this does not address the risk of fragmentation and diluted focus, nor the need for both the Bank and its clients to track results and learn from success and failure. Second, it would be important to use—and continuously refine— implementation and results monitoring systems that would track progress toward achieving the outcomes in the results framework of the CPS and more tightly link knowledge activities with CPS milestones and outcome indicators. 5.26 A RAS program could be enhanced if it contained a consultation mechanism between the Bank and the authorities, which would include the identification of results indicators at the inception of the tasks, and a follow-up review of the actions taken and results achieved. Problems related to absence of strong enough results frameworks are often exacerbated by the high level of client staff rotation within public services, which complicate the tracking of knowledge service recommendations and their implementation. Moving toward programmatic knowledge services should facilitate tracking of results over time, notably in areas with multiyear and interconnected Bank support. More generally, arrangements could be explored for sharpening the strategic relevance of rapidly evolving RAS programs. Options could include the establishment of funds cofinanced by the authorities and the Bank, with the possible participation of trust funds, with the aim of prioritizing programmatic knowledge services, knowledge services on important issues for the medium-term development agenda, or TA to lagging areas of the country. How Can the Bank Group Leverage Its Engagement in Knowledge-Based Country Programs? 5.27 The synergy of World Bank knowledge activities and IFC Advisory Services projects could be strengthened. Developing robust results frameworks in country partnership strategies that articulate the CPS outcomes and their linkages with the Bank Group’s programs and instruments is a key factor for enhancing the synergy between World Bank knowledge services and IFC Advisory Services in countries with World Bank knowledge-based partnerships. Synergy would also be enhanced by conducting core knowledge services for private and financial sector development—such as ICAs and FSAPs. Beyond the current practice of informal coordination between staff in country offices, there may be merit to establishing a 74 CHAPTER 5 CONCLUSIONS formal mechanism of including each institution in the other’s review processes. Development of joint programs is another option to consider; this has been done in other countries. Joint projects—as in the Subnational Doing Business project and land registration projects in the Russian Federation—could also be used more often. 5.28 Improving the way the Bank learns from upper-middle-income countries and intermediates this knowledge to other countries. The Bank’s engagement in countries with knowledge-based programs generates relevant knowledge for other, especially neighboring, countries. There is scope for improving the way the Bank intermediates knowledge generated through knowledge-based country programs. Exchanges could be enhanced within the Bank through communities of practice and outside the Bank through networks of practitioners. The recently launched Public Expenditure Management Network for Asia (PEMNA) is a good example of such a peer-based learning network of practitioners in public financial management. Portals on sector work in country office websites would also help share experiences. The role of the regional chief economists would also have to be enhanced, beyond the supervision of regional flagship reports, to linking up with regional think tanks and research institutions that can play a role in sharing the Bank’s analytical work in knowledge-based partnerships across countries. Indeed in countries with a preponderance of knowledge services this role should not be assigned to a single position or unit, but chief economists are well positioned to take the lead and coordinate the linking work in knowledge-based partnerships. Easing the confidentiality of knowledge services conducted through RAS would also facilitate knowledge transfers. Steps in this direction suggested by a Bank review of fee-based services could somewhat alleviate this constraint. 5.29 More generally, technical capacity developed by upper-middle-income country clients could be leveraged more effectively in Bank knowledge work. Thailand, for example, has built capacity in banking, payments system, and financial markets. The Bank could leverage this capacity in other countries, rather than bringing in a large number of international consultants, which will also provide an opportunity for the Bank to partner with Thai institutions. The same applies to Malaysia’s and Chile’s experience, which is of great interest respectively to other Association of Southeast Asian Nations and Latin American countries. Through its regional hubs and staff working on neighboring countries, the Bank informally transmits, to some extent, to other countries the knowledge gained in countries such as Chile, Malaysia, and Thailand. However, more can be done as there is a sense that these South-South exchanges remain weak. 5.30 Table 5.1 summarizes the links between findings, lessons, and recommendations discussed in this report. 75 Table 5.1. Findings, Lessons, and Recommendations IEG Findings IEG Lessons IEG Recommendations Achievement of results from knowledge services Attention to the relevance and technical Conduct broad consultation (with the client and other stakeholders) was driven by their relevance and technical quality of knowledge services is essential for on the issues to be addressed; deploy highly experienced staff with quality, including task designs that responded results—regardless of how knowledge global perspective and ability to deliver analytical and advisory to client concerns; customization of international services are financed—and for shaping activities (AAA) products on time; and adhere to the mandatory best practice to local conditions, including learning under the “science of delivery” that is AAA quality assurance process. Encourage emerging knowledge capacity constraints; generation of data to central to the present Bank reform effort. hubs to follow approaches along these lines. support policy making; formulation of actionable recommendations that fit local administrative and political economy constraints; and timely delivery of products to influence key decisions (par. 4.15). Most tasks referred to the local policy context, Understanding the political economy of reform Involve local experts, partners, and local knowledge hubs more albeit in variable detail. Those that achieved and using local expertise can enhance the extensively in AAA to help understand better the political economy results probed more deeply into local context impact of the Bank’s AAA. Local partners or of reform in the country where advice is sought, bridge the gap (see pars. 3.9-3.13) and used local expertise hubs can also be critical in conveying relevant between international good practices and local conditions, enhance more often than those that did not achieve country context considerations. the applicability of the recommendations, and build the local results. capacity to achieve longer-term impact. Tasks that achieved results provided actionable Emphasizing the “how to” options, as opposed Give staff more time to interact with clients and local partners and recommendations more often than those that to the diagnostics and the “what to do” knowledge hubs, including through adequate field presence; did not achieve results. recommendations, will enhance a client’s deploy available analytical expertise to ensure high-quality ability to own policy decisions, action plans, or research underpinning recommendations and high-level consultant strategies. expertise able to provide practical know-how and enable customization of global practice. Use of programmatic approaches was Staying engaged and responsive in the Design programmatic AAA in a number of thematic areas (for important to achieve outcomes. There are implementation phases of Advisory Services example, public financial management) that build on initial work to examples of programmatic knowledge services through appropriate instruments can help support implementation phases, including engagement of a broad that enabled technical assistance (TA) or clients translate recommendations from sound range of stakeholders to help disseminate the reform agenda and economic and social work (ESW) to follow up on analysis into actions that fit local political and maintain the focus on key policy issues in the public domain. initial work (par. 5.14). In contrast, some administrative constraints. 76 CHAPTER 5 CONCLUSIONS Table 5.1. Findings, Lessons, and Recommendations IEG Findings IEG Lessons IEG Recommendations projects launched in response to demand did not produce strong results, sometimes because these were one-off initiatives with poor sustainability prospects (par. 4.20). Complementing knowledge services with Close links between Bank ESW, nonlending Design country partnership strategies (CPSs) with closer links lending made results more likely to be TA, and projects may help sustain results. between knowledge activities and lending, including programmatic sustained. series deploying both instruments to support the paths from consideration of policy options to implementation of selected approaches. Linking Bank with International Finance Stronger synergies between World Bank Conduct core knowledge activities for private and financial sector Corporation (IFC) activities also helped knowledge services and IFC Advisory development and develop joint Bank/IFC programs and projects achieve results (see examples in pars. 4.23- Services projects can improve results. within CPS results frameworks that articulate the outcomes and 4.29). But the experience with Bank-IFC their linkages with the programs and instruments of both coordination has been mixed, with quality of the institutions. This would be complemented by formal mechanisms of CPS results framework and the existence of including each institution in the other’s review processes. core ESW among the factors that influenced the degree to which World Bank and IFC knowledge services had synergy. The capacity to see the big picture and provide Remaining engaged in areas that are relevant Prepare CPSs with countries where knowledge-based country multisectoral development solutions has been a to countries’ medium-term development programs are being implemented without a CPS (which does not strength of the Bank’s knowledge services, agenda may help the Bank maintain its have to follow a burdensome consultation process) to provide generally valued by clients. Delivery of capacity to see the big picture and provide guidance on Bank Group engagement objectives and avoid knowledge services through a “consultant firm multisectoral development solutions. fragmentation of knowledge activities away from evolving model,” which reflects a drive to accommodate development priorities. Use instruments (such as high-level multiple and unforeseen needs, often results in brainstorming, conferences, and ESW, including periodic Economic fragmentation of Reimbursable Advisory Monitoring Reports) and commit the necessary resources to Services (RAS) programs (for example, by identify, follow up, and sustain emphasis over time on issues that dropping tasks linked to medium-term objectives are important for medium-term development. to accommodate shorter-term needs) and may 77 CHAPTER 5 CONCLUSIONS Table 5.1. Findings, Lessons, and Recommendations IEG Findings IEG Lessons IEG Recommendations dilute the focus on important medium-term issues. Client participation in the different stages of Conducting a broad consultative process and Broaden the participation of various stakeholders and local knowledge activities appears to be closely dissemination, acknowledging the public good knowledge hubs into knowledge-based country programs (for associated with success in achieving expected function of Bank knowledge services, while example, by opening up discussions or focus groups with local knowledge service outcomes. paying attention to local circumstances, may experts and CSOs) and make Bank studies more widely accessible improve results. (for example, by recognizing upfront the public good component in knowledge products and sharing a portion of the knowledge service cost with the client on the condition of its disclosure). Bank knowledge activities were not monitored Implementation and results monitoring Use—and continuously refine—implementation and results and evaluated consistently in the sample of systems are needed to track progress toward monitoring systems that would track progress toward achieving the countries (par. 4.22). Where it was, better mutually agreed outcomes and mitigate the outcomes in the results framework of the CPS and that more tightly knowledge service results were more likely to be risk of fragmentation and loss of strategic link knowledge activities with CPS milestones and outcome achieved, probably reflecting a link between focus intrinsic in RAS. indicators. A “circle of continuous quality improvement” of M&E is monitoring and evaluation (M&E), knowledge critical for shaping the science of delivery that the Bank is presently service quality, and impact. intent upon and to help improve M&E at the country level. There were ample opportunities for learning There is scope to improve the way the Bank Enhance exchanges of knowledge within the Bank through from development experiences in the focus learns from upper-middle-income countries communities of practice and outside the Bank through networks of countries (for example, on development and intermediates this knowledge to other practitioners or knowledge hubs; enhance the links of the Bank’s trajectory from low-income to upper-middle- countries. regional chief economists with regional institutions that can play a income economy in Malaysia or the extensive role in sharing the Bank’s analytical work; ease the confidentiality work the Bank has done in China on regional of knowledge activities conducted through RAS; and leverage the approaches to investment promotion). technical capacity developed by upper-middle-income clients to other countries. This will also provide opportunities for the Bank to strengthen partnerships with local institutions in countries with knowledge-based programs. Although the relevance of RAS is strengthened Although RAS do not seem to enable Move decisively toward RAS in knowledge-based programs to by client demand and financial commitment, the knowledge activities to get more traction on sustain this business line, while clarifying the types of knowledge achieved results do not appear significantly results, they help expand the feasible set of products that come close to “public knowledge goods” (Bank- 78 CHAPTER 5 CONCLUSIONS Table 5.1. Findings, Lessons, and Recommendations IEG Findings IEG Lessons IEG Recommendations different from those of knowledge services services it provides to countries, ensure the funded reports targeted to a broad audience to disseminate funded by the Bank’s own resources. Other sustainability of the Bank’s business model in analyses of developments or particular sectors or issues), as fundamental factors—related to the relevance of knowledge-based country programs, and opposed to those that serve specific needs of counterparts. RAS design, quality, timeliness, client participation, generate new knowledge that the Bank can could be offered to institutions that can cover the full cost of the and use of local expertise—are more closely then intermediate to lower-income countries. Bank’s services, with cost-sharing of knowledge products that are associated with the achievement of results. not public knowledge goods in recognition of the relevance of the activities for other countries. In those products where there is cost- sharing, there should be wider disclosure or dissemination by the recipient for equalization of access to the Bank’s knowledge services. Full cost recovery could include the cost of a client survey, with the aim of informing both the Bank and the client about the relevance, quality, use, and results achieved or likely to be achieved. 79 Appendix A Rankings of World Bank Country Programs Table A.1. Programs Ranked by Lending and Analytical and Advisory Activities (AAA) Indicators — Including Associated IFC Advisory Services and Investments (FY05–11) Fee based Number AAA cost service in % of in % of TOTAL IFC IFC of country lending/ Lending country number of Advisory investments service Average number cost/ Ran service AAA FY05- Services /Advisory budget (FY05- Country rank of AAA Rank AAA cost k budget Rank 11 Rank total Services 09) Saudi Arabia 5 0.0 1 0.0 6 89 4 109 7 6 1.5 77.1 Russian 13 0.1 13 0.6 19 57 17 128 4 71 2.11 9.3 Federation Kuwait 15 0.0 2 0.0 7 87 5 41 44 44.5 Thailand 18 0.2 25 0.5 16 58 14 74 15 6 1.3 4.5 Kazakhstan 19 0.1 18 1.1 29 47 25 128 5 5 3.6 45.6 South Africa 22 0.1 16 0.6 20 50 24 56 28 26 1.0 18.9 Malaysia 23 0.0 10 0.1 10 82 9 31 61 36.4 Indonesia 24 0.6 52 0.7 23 56 19 215 1 54 0.9 Algeria 24 0.1 11 0.5 15 58 15 34 56 12 0.7 36.4 Syrian Arab 26 0.2 24 0.1 8 83 7 30 63 2 1.0 Republic Egypt, Arab 27 0.4 28 1.3 34 41 34 99 10 30 1.3 1.0 Rep. Morocco 28 0.4 34 1.2 31 41 33 87 12 26 0.4 Bahrain 28 0.0 3 - 1 92 3 15 105 59.5 Qatar 32 0.0 5 - 2 93 2 12 117 92.0 Chile 32 0.4 29 1.3 35 41 31 49 34 2 9.5 14.5 United Arab 33 0.0 6 - 3 96 1 10 123 7 0.1 72.8 Emirates Zimbabwe 34 0.2 21 0.1 9 63 13 18 92 1 0.0 Zambia 35 0.4 38 1.4 39 38 38 59 26 8 1.1 Oman 36 0.0 7 - 4 86 6 9 125 4 1.5 203.5 Pacific Islands 36 0.2 22 0.5 17 53 21 21 82 Philippines 37 0.6 55 1.5 45 38 40 100 9 24 0.9 Libya 38 0.0 8 - 5 82 8 6 130 79.3 Swaziland 38 0.2 20 0.6 21 56 18 18 93 1 0.0 Sudan 39 0.7 78 1.0 28 45 27 60 24 9 0.0 Pakistan 39 0.7 75 1.3 33 39 36 81 13 35 1.5 Lithuania 40 0.0 4 0.7 22 57 16 13 116 1 1.0 Latvia 41 0.2 19 0.9 26 51 23 18 95 4 0.0 8.9 Timor-Leste 41 0.5 45 1.4 40 39 35 42 43 3 0.0 Namibia 41 0.3 26 0.7 24 53 20 18 94 1 2.0 Turkey 41 0.5 46 1.7 54 34 50 80 14 3 21.3 Ukraine 42 0.4 30 1.7 56 34 51 54 30 23 2.0 Belarus 42 0.5 43 0.9 25 46 26 26 74 7 3.1 Mauritius 43 0.4 35 0.9 27 44 28 22 80 2 0.5 Mongolia 43 0.6 56 1.5 44 38 41 54 29 10 2.1 Fiji 44 0.1 17 0.5 18 65 11 7 128 11 0.1 Cambodia 44 0.4 39 1.8 58 32 61 70 17 29 0.3 Poland 45 0.4 36 1.5 47 38 39 33 58 5 0.2 Iraq 46 0.5 42 1.6 51 29 72 68 18 4 2.5 Slovak Republic 46 0.1 15 1.2 32 42 30 15 107 1 81 APPENDIX A RANKING OF WORLD BANK COUNTRY PROGRAMS Fee based Number AAA cost service in % of in % of TOTAL IFC IFC of country lending/ Lending country number of Advisory investments service Average number cost/ Ran service AAA FY05- Services /Advisory budget (FY05- Country rank of AAA Rank AAA cost k budget Rank 11 Rank total Services 09) West Bank and 46 0.7 66 1.4 43 36 45 53 31 6 1.5 Gaza Equatorial 47 0.3 27 0.4 13 63 12 3 137 Guinea Nepal 48 0.7 70 1.4 42 36 44 49 35 11 0.9 Czech Republic 48 0.1 12 1.4 37 41 32 14 112 23 2.0 Solomon Islands 50 0.4 33 1.1 30 43 29 15 106 17 0.1 Serbia 50 0.7 61 1.6 52 35 48 46 38 19 1.3 Jordan 51 0.4 41 1.8 57 33 53 38 51 16 1.2 Botswana 53 0.4 32 1.4 41 38 42 18 96 1 4.0 China 53 0.7 62 2.2 77 29 71 205 2 70 2.1 0.12 Seychelles 53 0.6 51 0.5 14 53 22 9 126 1 2.0 Iran, Islamic 54 0.1 14 1.8 60 31 63 25 78 1 3.0 Rep. Bangladesh 54 0.6 58 2.1 74 29 73 89 11 61 0.3 Kosovo 54 0.6 57 1.7 55 35 47 34 57 Somalia 55 0.8 82 0.2 11 68 10 12 118 Mexico 56 0.9 100 1.6 53 34 52 66 20 16 4.4 6.30 Madagascar 57 0.5 50 1.9 65 29 70 44 41 22 0.6 Maldives 57 0.5 44 1.4 38 32 55 19 90 8 0.6 Nigeria 57 0.8 83 1.9 68 32 62 74 16 17 2.8 Uzbekistan 58 0.4 37 2.0 71 30 65 32 59 13 0.5 India 59 0.7 65 2.4 85 27 84 194 3 67 2.7 Bulgaria 61 0.4 31 2.4 83 28 78 39 50 5 2.6 Uruguay 61 0.6 53 1.9 64 32 56 28 70 1 6.0 1.61 Lebanon 63 0.6 60 1.9 63 31 64 30 64 12 1.2 Malawi 64 0.6 54 2.2 76 28 80 41 45 1 7.0 Yemen, Rep. 65 0.7 67 2.4 86 27 83 63 22 10 0.6 Peru 65 0.9 98 2.0 70 32 59 53 32 26 2.2 Tunisia 65 0.5 49 2.4 84 28 81 41 46 3 2.3 Rwanda 66 0.9 95 1.8 59 32 58 38 52 19 0.6 Vietnam 66 0.9 91 2.4 88 28 79 124 6 29 1.4 Uganda 68 0.9 96 2.0 72 30 67 48 36 9 1.6 Sri Lanka 68 0.8 80 2.0 73 28 77 44 42 8 1.6 Togo 68 1.1 109 1.3 36 36 46 22 81 Romania 70 0.4 40 2.8 102 25 96 45 40 6 5.0 Tajikistan 70 0.8 89 2.2 78 29 74 45 39 21 0.7 Lao PDR 70 0.7 76 2.5 90 25 94 64 21 7 1.1 Turkmenistan 71 0.5 48 1.5 48 33 54 4 133 Liberia 72 1.5 130 1.5 46 37 43 28 69 11 0.6 Burundi 73 0.8 90 1.9 67 30 68 29 67 Paraguay 74 0.7 72 2.2 75 30 66 21 83 Afghanistan 74 0.9 99 2.5 91 26 88 67 19 9 1.0 Niger 76 0.9 101 1.9 66 32 60 26 75 4 0.5 Kenya 78 0.7 63 2.9 109 23 111 58 27 37 0.9 Tanzania 80 0.8 85 2.9 107 23 104 61 23 10 1.9 Côte d'Ivoire 81 1.0 104 1.8 61 28 76 21 84 Albania 82 0.7 77 2.8 103 25 93 36 54 20 0.4 Congo, Rep. 82 0.6 59 2.3 79 25 99 19 91 Brazil 83 1.2 116 2.9 104 24 102 101 8 22 4.9 1.59 Kyrgyz Republic 83 1.0 105 2.6 94 27 86 41 47 4 4.0 82 APPENDIX A RANKING OF WORLD BANK COUNTRY PROGRAMS Fee based Number AAA cost service in % of in % of TOTAL IFC IFC of country lending/ Lending country number of Advisory investments service Average number cost/ Ran service AAA FY05- Services /Advisory budget (FY05- Country rank of AAA Rank AAA cost k budget Rank 11 Rank total Services 09) Bhutan 84 0.8 84 2.4 87 25 98 29 68 12 0.2 Mozambique 85 0.7 79 3.0 112 22 115 53 33 17 0.5 Georgia 85 1.1 114 2.3 80 28.6 75 27 72 8 2.6 4.52 Central African 86 1.6 133 1.5 50 32.3 57 16 102 Republic Angola 86 0.5 47 2.6 92 23.0 107 18 99 1 4.0 Congo, Dem. 86 1.2 117 2.3 81 26.7 85 31 62 11 1.0 Rep. Djibouti 87 1.1 111 2.0 69 29.6 69 18 97 Bolivia 87 0.7 69 3.0 110 23.4 103 30 65 15 0.5 Mali 87 0.8 86 2.9 105 23.8 101 36 55 6 1.2 Gabon 89 0.7 74 1.9 62 26.1 89 7 129 2 0.5 Moldova 89 0.9 94 2.9 108 23.9 100 38 53 Ethiopia 89 0.7 68 4.7 133 16.7 130 60 25 Bosnia and 90 0.8 88 2.7 98 25.0 95 26 77 18 0.89 Herzegovina Azerbaijan 90 0.8 81 3.3 118 22.2 114 40 48 12 2.3 Mauritania 90 0.9 92 2.6 93 25.5 91 21 85 1 4.0 São Tomé and 92 2.5 139 1.5 49 34.7 49 4 132 1 1.0 Principe Gambia, The 93 0.7 71 2.9 106 22.8 108 21 87 Papua New 94 0.9 102 2.3 82 27.4 82 15 108 7 1.3 Guinea Colombia 94 1.1 115 3.1 113 22.7 110 47 37 14 3.6 Costa Rica 96 0.8 87 2.7 97 25.6 90 15 109 2 4.0 Sierra Leone 96 1.5 131 2.5 89 26.5 87 26 76 16 0.3 Hungary 96 0.2 23 3.4 121 22.5 113 9 127 1 1.0 Macedonia, FYR 96 1.0 103 2.8 100 25.3 92 20 89 14 0.1 Guinea 101 1.1 112 2.7 95 24.8 97 18 98 2 1.0 Burkina Faso 101 1.3 123 2.7 96 22.6 112 27 73 8 1.5 Lesotho 102 0.9 97 2.8 101 23.2 105 16 103 6 0.0 Montenegro 102 0.7 73 3.2 114 22.8 109 14 113 6 0.7 Senegal 103 1.0 108 3.4 122 20.4 120 32 60 4 2.8 Ghana 104 1.3 122 3.5 123 19.4 123 40 49 8 3.6 Chad 110 0.7 64 4.1 126 16.8 129 12 120 4 1.3 Cameroon 110 1.0 106 3.3 120 19.2 126 21 88 5 2.2 Haiti 110 2.0 138 3.0 111 23.1 106 21 86 4 2.5 Ecuador 114 0.9 93 3.8 125 19.6 121 14 115 5 2.0 Benin 115 1.3 124 4.1 128 16.8 128 23 79 1 2.0 Armenia 115 1.7 135 4.1 127 18.7 127 28 71 7 2.6 El Salvador 116 1.1 113 3.3 119 21.9 117 14 114 3 2.7 Argentina 118 1.9 137 5.2 134 14.7 134 30 66 3 15.3 0.88 Cape Verde 120 1.1 110 3.7 124 19.4 124 11 121 3 0.3 Guinea-Bissau 120 1.4 128 3.2 116 21.7 118 12 119 1 0.0 Venezuela, RB 121 1.0 107 3.2 117 19.5 122 3 138 1 2 Comoros 122 1.8 136 2.8 99 20.9 119 4 134 1 0.0 Panama 122 1.7 134 3.2 115 22.0 116 10 124 3 5.0 Nicaragua 124 1.6 132 4.7 132 15.6 133 18 100 10 1.7 Croatia 125 1.4 126 6.1 135 13.5 135 16 104 7 0.9 Dominican 125 1.4 127 4.6 131 15.9 132 15 110 Republic Honduras 126 1.5 129 6.3 137 12.8 136 18 101 7 2.1 1.77 83 APPENDIX A RANKING OF WORLD BANK COUNTRY PROGRAMS Fee based Number AAA cost service in % of in % of TOTAL IFC IFC of country lending/ Lending country number of Advisory investments service Average number cost/ Ran service AAA FY05- Services /Advisory budget (FY05- Country rank of AAA Rank AAA cost k budget Rank 11 Rank total Services 09) Guatemala 126 1.3 120 6.1 136 12.4 137 15 111 4 3.0 Tonga 127 1.3 119 4.2 129 19.2 125 4 135 7 0.1 Guyana 129 1.3 125 4.3 130 16.0 131 6 131 3 1.7 Jamaica 130 1.3 121 6.7 138 11.7 138 11 122 3 3.7 Samoa 133 1.3 118 9.4 139 9.3 139 4 136 9 0.3 AVERAGE 0.7 2.2 34.9 38.8 12 2 TOTAL 101.6 302.7 4,822 5,350 Source: World Bank Business Warehouse data. Note: AAA cost includes both Bank budget and trust funds. 84 Appendix B Criteria for Country Program and Knowledge Activity Selection World Bank Country Program Selection 1. The working concept of “knowledge-based country programs” for this evaluation is based on two broad functional categories of programs with the preponderance of knowledge activities: • Programs in countries where there is no pressing need for Bank financing and the partnership is thus designed with a strong focus on knowledge services. • Programs in countries where the Bank’s engagement revolves around nonlending activities because there is an insufficient foundation for a reform- oriented lending program. In some of these countries there is domestic political conflict or there are unsettled political conditions. In most of these countries the Bank remains engaged through analytical activities, to keep abreast of economic developments and poverty, to help governments implement reform programs that would establish more favorable development conditions, and to inform future operations. In most conflict-ridden countries, Bank activities are closely coordinated with the donor community and often funded by trust funds, which increases the size of knowledge activities in Bank country services budgets. 2. Country Policy and Institutional Assessment (CPIA) ratings over calendar years 2004-10 were used as a criterion to characterize country programs with high relative analytical and advisory activities (AAA) intensity according to these two categories. Programs in countries with below-average CPIA ratings (below 3) were put in the second category, where the engagement revolves around knowledge services due to an insufficient foundation for a reform-oriented lending program. 3. To further circumscribe the working concept of knowledge-based programs (KBPs) for this evaluation, the focus was placed on the first half of 138 country programs categorized on the continuum of relative knowledge intensity (see Appendix A), after removing the programs in countries in the second category. Using this additional criterion, 48 of the 69 country programs in the first half of the continuum of relative knowledge intensity fit a working definition of KBPs for the purpose of this evaluation (see Appendix C). Focus countries for the evaluation have been selected from among the 48 countries that make relatively more intensive use of Bank core knowledge services, based on the following criteria: 85 APPENDIX B CRITERIA FOR COUNTRY PROGRAM AND KNOWLEDGE ACTIVITY SELECTION • Primary criterion: Focus on upper-middle-income countries by per capita income level over 2005-10. • Secondary criteria have been applied to include countries: ◦ mostly toward the high end of the knowledge service intensity continuum ◦ with diversified economic structure ◦ with above-average size of knowledge service program ◦ with no Bank lending services ◦ with significant (average or above-average) use of Bank lending services ◦ with fee-based services ◦ that are OECD members or new EU members 4. The application of these criteria for the selection of focus countries is indicated in Appendix D. The selected focus countries are Bulgaria, Chile, China, Kazakhstan, Kuwait, Malaysia, Russian Federation, South Africa, and Thailand. Selection of World Bank Knowledge Activities and IFC Advisory Services 5. Economic and sector work (ESW), technical assistance (TA), and IFC Advisory Services were selected for the sample with attention to links to the strategic priorities in the country partnership strategy of each focus country. The Country Management Units were also consulted. The sample includes knowledge services and Advisory Services delivered over FY05–11. IFC Advisory Services were reviewed in China, Kazakhstan, the Russian Federation, and South Africa where such services were present and where government agencies or other public entities were the recipients of the service. The sample consists of 266 knowledge products and 34 Advisory Services out of a total of 751 knowledge products and 185 Advisory Services projects (to both government and private sector recipients) delivered over FY05-11. Products with similar thematic focus were grouped in clusters of knowledge activities. The number of activities reviewed was 196 for Bank knowledge services and 32 for IFC Advisory Services. 86 APPENDIX B CRITERIA FOR COUNTRY AND KNOWLEDGE ACTIVITY SELECTION Table B.1. Distribution of Sample of Knowledge Activities and IFC Advisory Services across Countries and Thematic Sectors (FY05–11) Sample analytical Sample Economic Poverty and IFC policy/ reduction/ advisory Advisory Education/ Private public social activities Services Country Agriculture health sector Infrastructure sector protection (number) (number) Bulgaria 0 0 6 1 7 4 18 0 China 8 6 15 10 3 8 50 13 Chile 0 9 3 0 11 2 25 0 Kuwait 0 3 3 2 7 0 15 0 Kazakhstan 0 5 19 0 16 9 49 4 Malaysia 0 0 8 0 3 2 13 0 Russian 6 0 15 11 10 7 49 9 Federation South 6 0 11 0 4 0 21 8 Africa Thailand 0 8 7 0 9 2 26 0 Sample 20 31 87 24 70 34 266 34 Source: World Bank Business Warehouse data. 87 Appendix C Corporate and Economic Indicators Associated with 48 World Bank Programs with High Relative Knowledge Service Intensity (FY05-11) Number of lending/ Lending number of AAA cost in commitments analytical and Lending % of country Average GDP (IBRD +IDA + Average advisory cost/AAA service Number of per capita grants in US$ Country rank activities (AAA) cost budget AAA 2005-2010 million) Saudi Arabia 5 0.00 0.00 0.89 109 20,349.8 Russian Federation 13 0.12 0.63 0.57 128 13,553.3 916.2 Kuwait 15 0.00 0.00 0.87 41 49,245.3 Thailand 18 0.24 0.50 0.58 74 7,181.5 1,085.6 Kazakhstan 19 0.15 1.07 0.47 128 10,050.7 3,685.2 South Africa 22 0.14 0.64 0.50 56 9,227.8 3,766.3 Malaysia 23 0.03 0.15 0.82 31 12,462.7 Indonesia 24 0.60 0.72 0.56 215 3,480.8 14,389.9 Algeria 24 0.06 0.47 0.58 34 7,329.0 0.4 Egypt, Arab Rep. 27 0.36 1.30 0.41 99 5,047.0 5,685.0 Morocco 28 0.40 1.17 0.41 87 3,894.8 2,723.1 Bahrain 28 0.00 0.00 0.92 15 25,995.0 Qatar 32 0.00 0.00 0.93 12 72,362.4 Chile 32 0.37 1.31 0.41 49 12,974.5 228.6 United Arab 33 0.00 0.00 0.96 10 54,498.3 Emirates Zambia 35 0.44 1.38 0.38 59 1,267.0 574.0 Oman 36 0.00 0.00 0.86 9 22,815.0 Philippines 37 0.61 1.50 0.38 100 3,302.2 2,645.7 Pakistan 39 0.72 1.29 0.39 81 2,298.2 7,374.2 Lithuania 40 0.00 0.71 0.57 13 15,794.2 Latvia 41 0.17 0.92 0.51 18 14,263.5 568.6 Turkey 41 0.49 1.66 0.34 80 12,108.8 12,122.1 Namibia 41 0.33 0.73 0.53 18 5,595.8 20.3 Ukraine 42 0.37 1.75 0.34 54 6,114.7 3,112.2 Mauritius 43 0.41 0.92 0.44 22 11,224.7 323.7 Mongolia 43 0.61 1.50 0.38 54 3,314.8 265.1 Fiji 44 0.14 0.52 0.65 7 4,278.8 - Poland 45 0.42 1.51 0.38 33 15,767.3 5,696.3 Slovak Republic 46 0.13 1.19 0.42 15 18,829.5 8.0 Nepal 48 0.71 1.42 0.36 49 1,007.2 1,381.7 Czech Republic 48 0.07 1.36 0.41 14 22,135.8 30.0 Serbia 50 0.65 1.64 0.35 46 9,273.3 1,583.9 Jordan 51 0.45 1.75 0.33 38 4,856.0 619.2 88 APPENDIX C CORPORATE AND ECONOMIC INDICATORS ASSOCIATED WITH 48 WORLD BANK PROGRAMS WITH HIGH RELATIVE AAA INTENSITY (FY 05-11) Number of lending/ Lending number of AAA cost in commitments analytical and Lending % of country Average GDP (IBRD +IDA + Average advisory cost/AAA service Number of per capita grants in US$ Country rank activities (AAA) cost budget AAA 2005-2010 million) Botswana 53 0.39 1.41 0.38 18 12,113.0 621.1 China 53 0.65 2.22 0.29 205 5,449.8 12,365.1 Iran, Islamic Rep. 54 0.12 1.79 0.31 25 10,005.6 564.0 Mexico 56 0.91 1.64 0.34 66 12,506.5 15,857.1 Madagascar 57 0.55 1.92 0.29 44 894.7 911.0 Maldives 57 0.47 1.37 0.32 19 6,787.0 87.4 India 59 0.68 2.40 0.27 194 2,742.2 27,465.1 Bulgaria 61 0.38 2.39 0.28 39 11,079.8 1,037.5 Uruguay 61 0.61 1.89 0.32 28 11,176.0 940.3 Malawi 64 0.61 2.20 0.28 41 716.2 821.8 Peru 65 0.89 1.98 0.32 53 7,499.5 3,440.6 Tunisia 65 0.51 2.40 0.28 41 7,950.2 1,426.3 Vietnam 66 0.85 2.42 0.28 124 2,527.2 9,115.8 Rwanda 66 0.87 1.78 0.32 38 947.5 959.4 Uganda 58 0.88 2.05 0.30 48 1,042.3 2,602.1 Average 0.39 1.24 0.47 11,986.2 3,675.5 Total 2,781 a Gross domestic product (GDP) in purchasing power parity per capita in thousands of constant 2005 US$. 89 Appendix D Application of Criteria for the Selection of Focus Country Programs Selection criteria Primar y Secondary Focus Inclusion countries of Inclusion Inclusion of Inclusion Upper- close to the Inclusion countrie of countries of middle- high end of of s with countrie with countrie income the countries above s with no average or s among country “knowledge with average World above Presenc OECD or per services diversified size of Bank average e of fee- new EU capita preponderanc economie AAA lending lending/AA based member Country income e continuum” s program services A services s Saudi Arabia - - - - - - - - Russian Federation X X X X - X - Kuwait X X - - X - X - Thailand X X X X - - X - Kazakhstan X X X - - X - South Africa X X X - - - X - Malaysia X X X - X - X - Indonesia - - - - - - - - Algeria - - - - - - - - Egypt, Arab Rep. - - - - - - - - Morocco - - - - - - - - Bahrain - - - - - - - - Qatar - - - - - - - - Chile X X X - - - X X United Arab - - - - - - - - Emirates Zambia - - - - - - - - Oman - - - - - - - - Philippines - - - - - - - - Pakistan - - - - - - - - Lithuania - - - - - - - - Latvia - - - - - - - - Turkey - - - - - - - - Namibia - - - - - - - - Ukraine - - - - - - - - Mauritius - - - - - - - - Mongolia - - - - - - - - Fiji - - - - - - - - Poland - - - - - - - - Slovak Republic - - - - - - - - Nepal - - - - - - - - Czech Republic - - - - - - - - 90 APPENDIX D APPLICATION OF CRITERIA FOR THE SELECTION OF FOCUS COUNTRY PROGRAMS Selection criteria Primar y Secondary Focus Inclusion countries of Inclusion Inclusion of Inclusion Upper- close to the Inclusion countrie of countries of middle- high end of of s with countrie with countrie income the countries above s with no average or s among country “knowledge with average World above Presenc OECD or per services diversified size of Bank average e of fee- new EU capita preponderanc economie AAA lending lending/AA based member Country income e continuum” s program services A services s Jordan - - - - - - - - Botswana - - - - - - - - China X - X X - X - - Seychelles - - - - - - - - Iran, Islamic Rep. - - - - - - - - Bangladesh - - - - - - - - Kosovo - - - - - - - - Mexico - - - - - - - - Madagascar - - - - - - - - Maldives - - - - - - - - India - - - - - - - - Bulgaria X - X - - X Xa X Uruguay - - - - - - - - Malawi - - - - - - - - Yemen, Rep. - - - - - - - - Peru - - - - - - - - Tunisia - - - - - - - - Vietnam - - - - - - - - Rwanda - - - - - - - - Uganda - - - - - - - - a*Fee-Based Services initiated recently. Note: AAA = analytical and advisory activities; OECD = Organisation for Economic Co-operation and Development; EU = European Union. 91 Appendix E Knowledge Activities Reviewed World Bank Thematic Country Project Name Project ID Number Typeb Areac Bulgaria Accelerating Bulgaria's Convergence: The Challenge of Raising Productivity P101353 ESW EPPSG Bulgaria The Road to Successful European Union Integration: The Policy Agenda (Country Economic P078356 ESW EPPSG Memorandum) Bulgaria Leveraging EU Funds for Productivity and Growth P094806 ESW EPPSG Bulgaria Public Expenditures for Growth and Competitiveness P124703 ESW EPPSG Bulgaria TA on Performance-Based Budgeting and Public Investment Management P112458 ESW EPPSG Bulgaria Support for Justice System Reform; Judicial Public Exp and Inst Review P103047, P105636 ESW, TA EPPSG Bulgaria Bulgaria-Railways Policy Note P102022 ESW INF Bulgaria Poverty Monitoring - Policy Note P100659, P107715, P112824 ESW PRSP Bulgaria HD Policy Notes P113071 ESW PRSP Bulgaria Bulgaria Regulatory Reform - (Better Regulation Series) P114992 ESW PSD Bulgaria Corporate Governance P108490, P116719 ESW, TA PSD Bulgaria Investment Climate Assessment P100571, P104192 ESW, TA PSD Bulgaria Making Research and Innovation Work for Bulgaria P078931 ESW PSD Chile Tertiary Education P106874, P120842, P123150 ESW EH Chile Primary and Secondary Education P111889, P106750, P103453, P111888, ESW, TA EH P115620 Chile Early Childhood Development P116849 ESW EH Chile Chile Development Policy Review P084737 ESW EPPSG Chile Policy Assessment in the OECD: Lessons for Chile P115377 TA EPPSG Chile Chile: Regional Development Study P103331 ESW EPPSG Chile Decentralization P122526 TA EPPSG Chile Evaluation of the Evaluation System P094343 ESW EPPSG 92 APPENDIX E KNOWLEDGE ACTIVITIES REVIEWED Chile Management of Financial Assets P106653 TA EPPSG Chile Policy Assessment II P117291 TA EPPSG Chile Public Sector Management Modernization Agenda P106231, P117290, 117289, ESW, TA EPPSG 122527 Chile Implementation of Risk-Based Supervision Model for Insurance Industry P108612 TA FIN Chile Risk Based Supervision of Pension Funds P109002 TA FIN Chile Insurance Law Review P119023 ESW FIN Chile Programmatic DPR Poverty P094879 TA PRSP Chile Chile Policy Notes P095497 ESW PRSP China China Land Policy Reform for Sustainable Economic and Social Development P090454 ESW ARD China Integrating Land Policy Reforms II: Stren Land Rights for Equit Growth and Soc P096320 ESW ARD China Land Policy Reforms III P108701 TA ARD China The Economic Value of Water P096650 ESW ARD China Activity Completion Report: Irrigation and Resource Management P108915 TA ARD China Compliance with Food Safety Requirements for Fruits and Vegetables P085058 ESW ARD China Public Services for Building the New Socialist Countryside P101740 ESW ARD China Rural Public Services P108700 ESW ARD China Imp, M&E of Natnl Health Sys Ref; Africa-China Health Sys Stren; CN-Hospital P123539, P120151, P119332 TA EH China Noncommunicable diseases (NCD) TA P112220 TA EH China Rural Health Reform P082296, P115201 ESW EH China China-DAC study group and south-south learning P118379 TA EPPSG China China 2030 P127560 ESW EPPSG China Guangdong AAA - Reducing Inequality in Guangdong Province P111946 ESW EPPSG China China - Financial Sector Assessment Program (World Bank/IMF) P117143 ESW FIN China China - ROSC Accounting and Auditing P112572 ESW FIN China Access to Finance FY10 P117719 ESW FIN China China: Ensuring financial stability (Catastrophe Insurance) P110190 TA FIN China China: Financial Sector Assessment Program (FSAP) P117143 ESW FIN China China: Creditor rights, financial sector development, investment, and growth P090783 ESW FIN 93 APPENDIX E KNOWLEDGE ACTIVITIES REVIEWED China China: Capital Market (2): Preparation of Capital Market Development Report P099020 ESW FIN China China: Capital Market (3) P107660 ESW FIN China China: Capital Market (4): State Sponsored Venture Capital in China P112287 ESW FIN China China: ROSC - Accounting and Auditing P112572 ESW FIN China China: Low Carbon City P123996 ESW INF China China - Urban Transport (Building Institutions, 2006) P096830 ESW INF China China - Development Policy of the Railway P107631 TA INF China Air Pollution Control P108202 ESW INF China China - Energy 2020 P093919 ESW INF China China - Railway Restructuring P096735 ESW INF China China Road Safety Roadmap P121046 ESW INF China Road Traffic Safety P110301 ESW INF China China - Strategic Environmental Assessment P091513 TA INF China Strengthening Energy Efficiency P113833 TA INF China Outsourcing Social Services to CSOs: Lessons from Abroad P114022 TA PRSP China From poor areas to poor people: China's evolving P085127 ESW PRSP China Social protection and labor markets programmatic AAA Several (programmatic) ESW PRSP China The well-being of China's rural elderly and old age support P101704 ESW PRSP China China: Support for Sharing China's Development Experience P093554 TA PSD China China: 120 Cities Competitiveness and Governance Study P096703 ESW PSD China Innovation, R&D, and Competitiveness P102118 ESW PSD China Shanghai PSD and Innovation Program P113149 ESW PSD China China PSD Program P117716 ESW PSD Kazakhstan Education and Innovation Development (JERP) P104563 ESW EH Kazakhstan JERP Policy Dialogue P097536 TA EH Kazakhstan JERP Higher Education P114472 TA EH Kazakhstan JERP Higher Education P120177 TA EH Kazakhstan JERP Secondary Education P120943 ESW, TA EH Kazakhstan Debt Management Performance Assessment (DeMPA) P122536 ESW EPPSG 94 APPENDIX E KNOWLEDGE ACTIVITIES REVIEWED Kazakhstan Reforming Intergovernmental Fiscal Relations, Review of Gen Trans Calc Met P088990, P113411 ESW, TA EPPSG Kazakhstan Policy Advice on PAR, JERP PAR P105979, P113409 TA EPPSG Kazakhstan JERP Kazakhstan Small Business Taxation P121252 TA EPPSG Kazakhstan Tax Policy and Tax Administration P096940, P102815, P114164, ESW, TA EPPSG P111105 Kazakhstan WTO Accession Advice P083571, P104202 TA EPPSG Kazakhstan Growth and Compet in Kazakhstan: Macro, Indust, Trade, Inst Dev Policies P095545 TA EPPSG Kazakhstan Macro modeling TA and KER 1&2 P113888 TA EPPSG Kazakhstan JERP-Impr of Natnl Fund Mngt, Comp Rev Glob Exp Mngt Nat ResWealth Funds P119579 TA EPPSG Kazakhstan Kazakhstan: Public Expenditure and Fiduciary Accountability (PFMA) P107737 ESW EPPSG Kazakhstan Brainstorming on Financial Volatility P114772 TA FIN Kazakhstan ROSC Insolvency and Creditors Rights 2011 P123680 TA FIN Kazakhstan Brainstorming Anti-Crisis Measures P117130 TA FIN Kazakhstan Financial Sector Assessment Program Update P110451 ESW FIN Kazakhstan ROSC Accounting and Auditing P904972 TA FIN Kazakhstan Financial Sector Reform P092483 TA FIN Kazakhstan Financial Systems Enhancement-Fin Instruments for infrastructure P096612 ESW FIN Kazakhstan FSAP Follow Up P113817 TA FIN Kazakhstan Insolvency and Corporate Restructuring P112504 TA FIN Kazakhstan Pension Policy Note, Pension system support (JERP), Pension system (JERP) P080299, P102198, P108810, P116757, ESW, TA PRSP P119510 Kazakhstan Improvement of the social protection system P119511 ESW PRSP Kazakhstan Poverty Monitoring P112991 ESW PRSP Kazakhstan Survey of Quality of Government Services Study P111702 TA PRSP Kazakhstan Survey of Quality of Government Services - Health and Education P111933 TA PRSP Kazakhstan Competition Policy-Training, Inv Climate Assess, Consumer Protection Fin P113805, P113823, P120592, P123561, ESW, TA PSD P123668 Kazakhstan Doing Business I: Investor protection, Doing Business II, III: Overall Results P115729, P119266, P123638 ESW, TA PSD Kazakhstan Diversification of the Economy - Technoparks, Supply Chain Development P108783, P108938 TA PSD Kuwait Kuwait MOE: KEIAP Project FY04-06 (Education Expenditure Analysis Study) P078899 ESW EH Kuwait Kuwait MOE KEIAP; Kuwait KEIAP FY09-10 Support P101252, P112093 TA EH 95 APPENDIX E KNOWLEDGE ACTIVITIES REVIEWED Kuwait Country Procurement Assessment Review - Kuwait P097904 ESW EPPSG Kuwait Civil Service Reform - Kuwait P109695 ESW EPPSG Kuwait Governance and Integrity Program P115012, P105246 TA EPPSG Kuwait PEFA; Overview of Public Spending Allocations P115011, P115899 TA EPPSG Kuwait Kuwait MOF: Vision and Long Term Dev Options-National Strat for Eco Soc Dev P107345 TA EPPSG Kuwait Kuwait Bank Governance P120375 TA FIN Kuwait FSAP Update P121211 ESW FIN Kuwait Kuwait Economic Zone Development P100923 TA FIN Kuwait Kuwait: National Strategy for the Clean Development Mechanism (CDM) P123767 TA INF Kuwait Kuwait - Establish Land Transport Authority P122183 TA INF Malaysia MY-Brainstorming Higher Income and Growth P116627 TA EPPSG Malaysia MY: Penang's New Sources of Growth P104952 ESW EPPSG Malaysia Malaysia Economic Monitor (MEM) 2011 - MEM Brain Drain P122765 ESW EPPSG Malaysia Malaysia A&A ROSC P124592 ESW FIN Malaysia Malaysia-ROSC (Corporate Governance) P094644 ESW FIN Malaysia Optimal Design for a Minimum Wage Policy in Malaysia P125706 ESW PRSP Malaysia MY-Productivity Linked Wages System P122578 TA PRSP Malaysia Malaysia-Moving Up Value Chains P105787, P108571, P091782, TA PSD P124650 Malaysia Malaysia Productivity and Investment Climate Assessment Update P108488 ESW PSD Malaysia Malaysia-Value Chain Study P122583 TA PSD Russian RF-Water Quality Management Improvement P093645 TA ARD Federation Russian Russian Grain Reserves Study P116560 ESW ARD Federation Russian Russian Federation - Issues and Options of State Grain Trading P114108 ESW ARD Federation Russian RF-Farm Bill Review P087823 TA ARD Federation Russian RF-Agricultural Public Expenditure Review P083842 ESW ARD Federation 96 APPENDIX E KNOWLEDGE ACTIVITIES REVIEWED Russian RF-Agricultural Policy Monitoring P091861 TA ARD Federation Russian Russian Federation-Reshaping Economic Geography P112607 ESW EPPSG Federation Russian Russian Federation: Long-Term Fiscal Risks and Fiscal Sustainability P107247 ESW EPPSG Federation Russian A note on the efficiency of public expenditures in Russian Federation, Pub Exp Review P119392, P112597 ESW, TA EPPSG Federation Russian Performance-Based Budgeting TA, Performance-Based Budgeting TA 2 P092511, P097625 TA EPPSG Federation Russian Civil Service Reform TA, Civil Service Reform TA 2 P079300, P094428 TA EPPSG Federation Russian Regional Development Strategy P109701 TA EPPSG Federation Russian Country Economic Memorandum: Regional Development and Growth Agglomerations P076129 ESW EPPSG Federation Russian Russian Federation-Financial Sector Economic Memorandum- The Role of Finance for SME P116124 ESW FIN Federation Russian Crisis Response TA P102441 TA FIN Federation Russian Financial Sector Assessment Program Update P104863 ESW FIN Federation Russian Post FSAP TA Payments Systems: Challenges and Improvement Opportunities P108310 TA FIN Federation Russian Diagnostic Review of Consumer Protection in Financial Services P112491 ESW FIN Federation Russian Banking Sector Vulnerability Assessment P116343 ESW FIN Federation Russian Basel III Fin Architecture and Emerging Reg Dev in Macro Prudential Tools P123029 TA FIN Federation Russian Russian Federation - Energy Efficiency P096612 ESW INF Federation Russian Russian Federation-Past Environmental Liabilities P097152, P108003 ESW INF Federation Russian Russian Federation-Environmental Performance Review P108003 ESW INF 97 APPENDIX E KNOWLEDGE ACTIVITIES REVIEWED Federation Russian Russian Federation-Housing P093771 ESW INF Federation Russian Russian Federation -Policy Dialogue and TA on Public-Private Partnership P108022 TA INF Federation Russian Russian Federation-FBS Pulkuvo Airport P109593, P111493, P122493 TA INF Federation Russian Urban Transport, St. Petersburg P124324, P125148 TA INF Federation Russian Programmatic Poverty Work (Stage 2) P074995 ESW PRSP Federation Russian Subsidy Targeting for Households, Policy Note on Demography, HD Adv Serv P083408, P107778, P113017, ESW, TA PRSP Federation P118156 Russian Youth in the Russian Federation FY06, Labor Market Monitoring and Assessment FY10 P090671, P118448 ESW, TA PRSP Federation Russian Public Policies in Support of Competitiveness and Industrial Diversification P094380 TA PSD Federation Russian Russian Federation Doing Business Sub-national P108387 ESW PSD Federation Russian Export Diversification, Competition, and Innovation P108962 ESW PSD Federation Russian Economic Diversification P108164 ESW PSD Federation Russian Russian Federation Innovation P123690 TA PSD Federation Russian Nurturing Innovation Management in Russian Federation P125925 TA PSD Federation Russian Special Economic Zones P094821 TA PSD Federation Russian Investment Climate Assessment P089187 ESW PSD Federation South Africa Land Reform Support in South Africa P095688, P097247, P105553, ESW, TA ARD P105554, P107151 South Africa Fee Based Technical Assistance for Rural Development and Land Reform 1 P118654 TA ARD 98 APPENDIX E KNOWLEDGE ACTIVITIES REVIEWED South Africa Macro modeling P110934, P078974, P112925, ESW, TA EPPSG P119361 South Africa Financial Sector Assessment P114557, P118672, P121262, ESW FIN P098529, P112944 South Africa World Bank's Insolvency and Creditor Rights (ICR ROSC) P110211, P087423, P120937, ESW FIN P108461 South Africa Analysis of Investment Climate Variables P102408 ESW PSD South Africa Analysis of Investment Climate Constraints P113919 TA PSD South Africa Investment Climate Assessment P083285 ESW PSD South Africa Investment Climate Assessment Update P107001 ESW PSD South Africa PSD Activities (MSME) P090855 ESW PSD South Africa PSD TA Program P088615 TA PSD Thailand Social Monitoring Secondary Education P085871 ESW EH Thailand CDP Higher Education P106606 ESW EH Thailand CDP Health, Country Development Partnership in HD, Health P101768, P123383, P115330, ESW EH P117573 Thailand Thailand HIV/AIDS - Thailand-CDP Health (FY08) P083759, P106607 TA EH Thailand Thailand CDP on Governance and Public Sector Reform: Anti-Corruption TA P096704, P107881, P112450, TA EPPSG P118762, P122617 Thailand Thailand CDP on Governance and Public Sector Reform: PEFA Report P096704, P107881, P112450, TA EPPSG P118762, P122617 Thailand You P096704, P107881, P112450, TA EPPSG P118762, P122617 Thailand Thailand CDP on Governance and Public Sector Reform: Pub Sec Dev Civ Ser P096704, P107881, P112450, TA EPPSG P118762, P122617 Thailand Thai CDP on Governance and Public Sector Reform: TA on Procurement P096704, P107881, P112450, TA EPPSG P118762, P122617 Thailand Thai CDP on Governance and Public Sector Reform: Tax Policy and Tax Adm P096704, P107881, P112450, TA EPPSG P118762, P122617 Thailand Towards a Knowledge Economy in Thailand P104856 TA EPPSG Thailand Collaboration Forum for a Stronger Thailand P117458 TA EPPSG Thailand Thailand Economic Monitor FY10, FY11 P117844, P122632 ESW EPPSG 99 APPENDIX E KNOWLEDGE ACTIVITIES REVIEWED Thailand Thailand: Financial Sector Assessment (Thailand FSAP) P100998 ESW FIN Thailand Thailand: Financial Sector Assessment/Module SFIs/Thailand SFIs (FSAP Mod) P123884 ESW, TA FIN Thailand Thailand: Financial Sector Assessment (Insurance Regulations) P110589 TA FIN Thailand Thailand: ROSC (Accounting and Auditing) P100450 ESW FIN Thailand Thailand: CDP-FC P092233 TA FIN Thailand Leading with Ideas: Skills for Growth P068098 ESW PRSP Thailand Social Monitor on Youth P097213 ESW PRSP Thailand Thai Investment Climate and Productivity Report, Thai Investment Climate P082076, P106335 ESW PSD a Total number of questionnaires for the World Bank projects reviewed: 196. b Type: ESW = economic and sector work, TA = technical assistance. c Thematic area: ARD = Agriculture and Rural Development; EH = Environmental Health, EPPSG = Economic Policy and Public Sector Governance, FIN = Finance, INF = Infrastructure, PEFA = Public Expenditure and Financial Accountability, PRSP = Poverty Reduction/Social Protection; PSD = Private Sector Development. Note: AAA = analytical and advisory activities, CDP = country development partnership, CSOs = country partnership strategy, DAC = Development Assistance Committee, EU = European Union, FSAP = Financial Sector Assessment Program, HD = Human Development, IMF = International Monetary Fund, JERP = Joint Economic Research Program, M&E = monitoring and evaluation, MSME = micro, small, and medium enterprises, OECD = Organisation for Economic Co-operation and Development, R&D = research and development, ROSC = Report on the Observance of Standards and Codes, TA = technical assistance. Please spell out and add above in alpha order; DPR, PAR, KER, PFMA, MOE, KEIAP, MEM, MOF, MY, A&A, RF, FBS, ICR, SFI, FC. 100 APPENDIX E KNOWLEDGE ACTIVITIES REVIEWED IFC Country Project Name Project ID Type Business Number Lineb China Access to Business Credit - Secured Transactions 539924 Advisory A2F Service China Access to Business Credit 538624 Advisory A2F Service China Credit Reporting System Development in China 521645 Advisory A2F Service China China Housing Finance Advisory Program 550907 Advisory A2F Service China Leasing - China 533464 Advisory A2F Service China China Personal Bankruptcy Law Project Phase I 561106 Advisory A2F Service China SME Access to Finance 538700 Advisory A2F Service China IFC Advisory Service - Doing Business in China 552007 Advisory IC Service China Minister of Commerce (MOFCOM) China IP Support for Lesser Developed 543984 Advisory IC Regions Service China PSD Study - Quality and Sustainability Performance of Private Enterprise 539386 Advisory IC Service China Sichuan Cluster Development 540303 Advisory IC Service China Sichuan Recovery Investment Promotion Component 566267 Advisory IC Service China Sichuan Value Chain Analysis 541484 Advisory IC Service Kazakhstan Kazakhstan Leasing Development Project, O-EA-Primary Mortgage Institutions 522126, 547628 Advisory A2F Service Kazakhstan Quality Mgt. Cert & Business Services to Local SME, Ispat Kamat SME 507189, 507141 Advisory SBA 101 APPENDIX E KNOWLEDGE ACTIVITIES REVIEWED Resource Service Russian Russian Federation Insurance Contractual Law 562750 Advisory A2F Federation Service Russian Primary Mortgage Development 537485 Advisory A2F Federation Service Russian Advisory Support to Government on Securitization Program 537525 Advisory A2F Federation Service Russian Securitization Advisory TA - Russian Federal Service for Financial Markets 555105 Advisory A2F Federation Service Russian Subnational Doing Business in Russian Federation 555505 Advisory IC Federation Service Russian Legislative Reform of Locating Procedures 538767 Advisory IC Federation Service Russian Monitoring the Impact of Reform of Locating Procedures 538751 Advisory IC Federation Service Russian Pilot Intermediaries Survey 538754 Advisory IC Federation Service Russian AnWorkRussia 538779 Advisory IC Federation Service a*Total number of questionnaires for the IFC projects reviewed: 32. b Business Line: A2F= Access to Finance, IC = Investment Climate, SBA - Sustainable Business Advisory. Note: IFC = International Finance Corporation, PSD = Private Sector Development, SME = small and medium enterprises, TA = technical assistance. 102 Appendix F Evaluation Template IEG Evaluation of World Bank Knowledge-Based Country Programs Questionnaire for review of ESW and TA in focus countries Contents A. Activity Data B. Relevance (i) Coherence with the CPS (ii) Relevance of design (iii) Ownership (iv) Capacity building C. Technical Quality (i) Transfer of relevant knowledge (ii) Recommendations (iii) Staffing D. Results (i) Use of the activity (ii) Participation and process E. Sustainability of Results F. Lessons Learned 103 APPENDIX F EVALUATION TEMPLATE A. Activity Data Activity title: Activity type: ESW TA Sector Board: Initiation date: Completion date: Government counterparts involved in the preparation of the activity: Other counterparts (donors, private sector, etc.): Cost of activity: Funding source(s): Was the activity part of a programmatic AAA series? Was the activity prepared in support of lending operations? If yes please list the operations: B. Relevance Note: Please indicate how the activity’s relevance can be assessed against the following questions. Provide evidence in the comments section (based on desk reviews and/or interviews with clients, staff, other stakeholders) to substantiate the response. Yes Partly No NA B.1. Were the issues addressed by the Bank’s AAA identified as a development constraint or opportunity in the country partnership strategy (or in previous sector work, particular lending operations, evaluations, or donor coordination forums)? Comment: B.2. Was the activity designed with the aim of: • Informing policy dialogue • Raising stakeholder awareness • Contributing to the design of lending products • Contributing to the design of country partnership • Providing solutions to specific policy issues • Reviewing reform options in a sector 104 APPENDIX F EVALUATION TEMPLATE • Other Comment: B.3. Was the design of AAA appropriate to meet its objectives? Comment: B.4. Did the client request or commission this activity? Comment: (if yes, please list the main reasons) B.5. Were there any issues relevant to the client that the AAA product was not able to address? Comment: Other Comments on Relevance: [elaborate succinctly as appropriate] C. Technical Quality Note: Please indicate how the activity’s technical quality can be assessed against the following questions. Provide evidence in the comments section (based on desk reviews and/or interviews with clients, staff, other stakeholders) to substantiate the response. (i) Transfer of relevant knowledge Yes Partly No NA C.1. Did the activity convey international best practice and relevant examples from the region and other regions? Comment: C.2. Did the activity refer to the institutional and policy context in the country? Comment: C.3. Did the activity generate new evidence or data that inform analysis and decision making? Comment: C.4. Was the knowledge generated/transferred by the Bank’s AAA product comparable to that of similar activities (if any) undertaken by: • other official development partners • think tanks/universities • global consulting firms 105 APPENDIX F EVALUATION TEMPLATE Comment: C.5. Did the activity identify and make use of local expertise? (local organizations, south to south learning) in the planning/design, results analysis, report writing, dissemination … Comment: (ii) Recommendations C.6. Did the AAA provide actionable recommendations consistent with the findings and the conclusions? Comment: C.7. Did the activity discuss the capacity requirements and administrative feasibility of implementing the recommendations? Comment: Other comments on technical quality: [elaborate succinctly as appropriate] D. Results Note: Please indicate how the activity’s results can be assessed against the following questions. Provide evidence in the comments section (based on desk reviews and/or interviews with clients, staff, other stakeholders) to substantiate the response. (i) Use of the activity Yes Partly No NA D.1. Were the activity’s outputs delivered timely to inform important decisions? Comment: D.2. Were the outputs of the activity used to: • inform new laws or regulations • design/reform government programs • raise stakeholder awareness • build recipient capacity • create or improve institutions • help reach consensus for change • support the design of Bank lending operations • support Bank strategy formulation • support subsequent dialogue in the sector/topic 106 APPENDIX F EVALUATION TEMPLATE • other Comment: D.3. Were the outcomes sought at inception achieved? Comment: D.4. Was there a results framework embedded in the country partnership strategy (CPS) to allow measurement of the direct contribution of the activity to the development outcomes sought by the CPS? Comment: D.5. Did the activity include clearly measurable result indicators? Comment: (ii) Participation and process Yes Partly No NA D.6. Did a government agency or other stakeholders participate in the different stages of the activity: • scoping • design • implementation • analysis of results • formulation of recommendations • review and feedback • dissemination Comment: D.7. Was the product of the activity discussed with senior policy makers? Comment: D.8. Was the product of the activity presented in a workshop/conference/on the web? Covered in the media? Comment: 107 APPENDIX F EVALUATION TEMPLATE Other comments on results: [elaborate succinctly as appropriate] E. Sustainability of Results Note: Please indicate how the sustainability of results of the activity can be assessed against the following questions. Provide evidence in the comments section (based on desk reviews and/or interviews with clients, staff, other stakeholders) to substantiate the response. Yes Partly No NA E.1. Are the outcomes of the activity likely to be sustained? Comment: E.2. Was the Bank’s AAA complemented by other activities (ESW, TA, lending, other) to achieve the expected results? Comment: E.3. Did the outcomes of the activity contribute to strengthening/developing institutions Comment: E.4. Did the activity contribute to strengthening analytical and policy formulation capacity of the recipients? Comment: E.5. Was achievement of the agreed capacity development outcomes effectively monitored and lessons learned used by the bank or Government counterparts? Comment: Other comments on the sustainability of results: [elaborate succinctly as appropriate] F. Lessons Learned (open comment section) 108 Appendix G Perceived Strategic Positioning of the World Bank in the Focus Countries 1. In Kuwait, where the Bank delivers exclusively Reimbursable Advisory Services (RAS), the quality of Bank work compares favorably with global consulting firms in specific technical fields where the Bank has established a standard and widely recognized methodology (Country Assessment Procurement Review, Public Expenditure and Financial Accountability) or cutting-edge expertise (environment). Consulting firms such as McKinsey were considered much stronger in the strategic field, while the IMF or private consulting firms fare better on analytical work in the financial sector. The Tony Blair consulting firm made recommendations for education development; however, counterparts consider the quality of the Bank’s work in education at a higher standard. 2. In Malaysia, where a large part of the Bank’s knowledge services program is conducted as RAS, the Bank is broadly recognized as a credible and neutral partner, and its knowledge activities compare favorably with analytical work conducted by other international institutions (Asian Development Bank [ADB]) and consulting firms (Boston Consulting Group, McKinsey, Booze Allen Hamilton). The Bank’s work quality, especially in economic analysis, its breath of expertise in many areas, and its wide range of country experiences make it a highly respected partner. Moreover, through the Bank, the client expects to access the best global experts. Malaysia’s collaboration with the ADB is intensive through the Association of Southeast Asian Nations (ASEAN+3) initiatives. However, the Bank is seen as surpassing the ADB’s analytical capacity and expertise. Malaysian governments have frequently requested advice from global consulting firms (Boston Consulting Group and McKinsey in particular). However, the Bank’s neutrality is often appreciated in this respect—“the Bank will give us what we need, not necessarily what we want,” as one counterpart put it. 3. In Kazakhstan, where the client is funding up to 85 percent of the Bank’s knowledge services, the Bank has a unique position in delivering analytical and advisory activities (AAA) notably through the brainstorming sessions chaired by the Prime Minister, which have been running since 2002. The quality of the Bank’s AAA is often perceived as surpassing that of other development institutions. Global consulting firms are seen as quite good on specific niches, such as financial centers (Boston Consulting Group). Overall, as a counterpart put it, “the World Bank’s advice represents good value for money.” 109 APPENDIX G PERCEIVED STRATEGIC POSITION OF THE WORLD BANK IN THE FOCUS COUNTRIES 4. In Chile, the Bank generally compares favorably to other international institutions (Inter-American Development Bank, Organisation for Economic Co- operation and development [OECD]) and global consulting firms on the quality, relevance, and timeliness of its knowledge products. The Bank is often seen as surpassing the Inter-American Development Bank’s analytical capacity and relying less on consultants of variable quality. The OECD is recognized as providing valuable international comparisons based on first-quality data. Compared to the OECD, the Bank’s knowledge services in some areas are considered less responsive than in the past to client demand, and therefore of less relevance, especially in areas where the Chilean authorities aspire to converge on the standards established by OECD member countries (for example, in program budgeting and spending reviews). However, Chile engages with the OECD on economic and sector studies, mainly through channels of peer review and comparisons with other OECD member countries that differ from the Bank’s capability to produce on-demand studies on issues of interest to the authorities. The Bank has produced highly regarded joint studies with the OECD in education. 5. In Bulgaria, a member of the European Union, the Bank generally compares favorably to other international institutions (European Investment Bank, European Bank for Reconstruction and Development), think tanks, and global consulting firms regarding the quality, relevance, and timeliness of its knowledge products. The 2007 Country Economic Memorandum and the 2009 Policy Notes have served as inputs to Bulgaria’s National Reform Program for the implementation of the Lisbon Agenda and, later, the Europe 2020 Strategy. 6. In the Russian Federation, the Bank is seen as having a clear advantage in private sector development issues through the use of Investment Climate Assessment and Doing Business methodologies. Strengths are also perceived in certain financial sector areas, such as the payments system and regulatory and risk-based work. However, improving the timeliness of delivery, especially during a crisis when the situation is changing rapidly, was seen as an area where the Bank could improve. In some of these areas, especially in supervision and regulatory regimes, the International Monetary Fund (IMF) is viewed as more supportive. In human and social development, Bank competitors are better in some areas. For example, the United Nations is better prepared to deliver work on demographic issues and the World Health Organization in some health areas. On public financial management issues the Bank has a unique position as a partner of choice, while most other partners, including the IMF, are not approached for advice on this sensitive agenda. In practically all infrastructure sectors, other institutions also have carried out activities. In some areas, such as energy efficiency, the Bank’s knowledge activities were the most comprehensive, as they covered all sectors where energy efficiency could be improved, while studies by others 110 APPENDIX G PERCEIVED STRATEGIC POSITION OF THE WORLD BANK IN THE FOCUS COUNTRIES focused on particular areas. In agriculture and rural development the picture is mixed. The Bank’s advice was superior in public agricultural expenditure, donor coordination, river basin pollution monitoring, and likely also when reviewing legal frameworks (for example, the farm bill) as well as devising an investment program to strengthen rural financial markets. Its comparative advantage in other areas was far less apparent. 7. In Thailand, where knowledge activities are funded from the Bank’s budget, knowledge services compare favorably to knowledge services by other development agencies and commercial consulting firms. The IMF focuses on short-term stability issues, but many of Thailand’s problems are structural, and therefore in the Bank’s domain of expertise. The Bank is seen as possessing stronger in-house expertise than the Asian Development Bank. The Bank’s advice is also seen as more credible than that of the International Labor Organization, which also has been active in social protection and labor. 8. In China, the Bank continues to enjoy high prestige for its analytical rigor, its cutting-edge expertise in areas of interest to Chinese policy makers, its neutrality and credibility, and its understanding of China accumulated over time. The relevance of knowledge activities is universally high, even when the Bank initiated the activities. Timeliness has sometimes been a problem. In fast-changing China, a report released two years after data collection, as it was the case with several knowledge products reviewed by the Independent Evaluation Group, is considered dated with limited policy guidance value. The government views contributions from other agencies (the United Nations, other donors, or global consulting firms) as complements to the Bank’s knowledge activities if the Bank takes a holistic, fundamental, nonconfrontational approach to the subject, using evidence-based analysis, drawing upon its global experiences. 9. Compared to the other focus countries, in South Africa the Bank has faced a more challenging environment for its strategic positioning as a knowledge provider. It is highly recognized for its technical capabilities and global experience. However, some counterparts question the Bank’s relevance owing to perceived too close association with the “Washington consensus.” As a result, the Bank’s ability to enter into policy dialogue and to influence reform can be highly constrained. The forums provided by the OECD to share best practice and learn from each other are examples of the type of dialogue that is much appreciated in South Africa. The United Nation Institute for International Development Research and the Harvard Development Institute were also cited by government counterparts as organizations with high-quality contributions and an open-minded approach. In South Africa, the Bank is also compared to highly 111 APPENDIX G PERCEIVED STRATEGIC POSITION OF THE WORLD BANK IN THE FOCUS COUNTRIES competent domestic consulting firms and academicians who have been providing tailored services to the government. 112 Appendix H Examples of Successful Knowledge Activities A. Leveraging the Bank’s Global Expertise: Chile Primary and Secondary Education 1. Although schools in Chile made progress on learning outcomes, as reflected in PISA (Program for International Student Assessment) scores from 2000 and 2006, the levels attained remained well below those achieved in Organisation for Economic Co- operation and Development [OECD] countries and that Chile aimed to attain. In 2006, protests from the secondary student community sharpened government attention to quality assurance in primary and secondary education. As a result, the government announced reforms in 2006 that included the creation of a new Education Oversight Institution and agreed with the Bank on fee-based support for three tasks: • Review models of how other countries distributed roles between education institutions, with focus on supervision, enforcement, policy-setting, and support functions • Review the institutions responsible for carrying out these functions, the instruments they use, and the way they relate to other agencies • Identify issues and guidelines for redesigning the Ministry of Education, with special emphasis on quality assurance. 2. The Bank provided advice on these matters during 2006-09 that illustrates the contribution that its capacity to deploy global knowledge can make. This advice is reflected in its final 2009 report to the government. In agreement with the government, the Bank selected for comparison with Chile education systems in Ireland, the Netherlands, New Zealand, and the United Kingdom (Scotland). This selection met two criteria: central government empowerment for key education policy decisions with decentralized service delivery and high performance in international learning tests. The international comparison focused on how the selected systems discharged eight functions: setting of performance standards, performance assessment, performance reporting, program and policy evaluation, setting of licensing requirements, provision of technical and pedagogic assistance, provision of adequate and equitable financing, and application of performance-based accountability. The comparison included the lessons from reforms processes in the systems covered. 3. The international comparison was customized to refer to Chile’s own system. The report further benefited from input from local consultants and from a close working relationship with the Ministry of Education over a period of about three years. Accordingly, the report provided recommendations primarily as lessons from 113 APPENDIX H EXAMPLES OF SUCCESSFUL KNOWLEDGE ACTIVITIES international experience on quality assurance and Chile’s experience on institutional reforms in those other sectors in Chile. By all accounts, the report was successful in assisting reform in Chile. It was used in developing the new quality assurance system for primary and secondary education, covering new legislation and institutions. As the new reform in Chile was established only recently, it is too early yet to assess its development outcome. B. Customization of Solutions to Country Context: Thailand’s Economics of AIDS Report 4. Despite its remarkable progress in AIDS prevention and control, Thailand’s projected numbers of HIV-infected persons still remained massive at the middle of the first decade of the millennium. The Bank was asked to provide technical support on several issues: • Most-at-risk-population (MARP) size estimation; design and implementation of state-of-the-art injecting drug use harm reduction policies and interventions • MARP costing, operational, contracting and performance management guidelines for effective contracting of targeted interventions for MARPs • Impact evaluation and cost-effectiveness analysis guidelines and antiretroviral therapy (ART) program performance assessment. 5. The Bank’s 2006 report The Economics of Effective AIDS Treatment, part of package of Bank AIDS/HIV analytical and advisory activities (AAA) support delivered during 2005-08, addressed these objectives in part through an analysis of the costs and benefits of alternative policy scenarios. The report is a good example of customization to country context. It thoroughly covered the institutional and policy context, provided new information with the first available estimation of the cost and expected benefits of ART, and was co-authored by local experts from the health policy and planning unit. On country context, the study documented Thailand’s remaining high prevalence rates among certain groups, most importantly the injecting group users; the changing sources of transmission, from infection in sex workers and their clients to infections in women by their husbands or sex partners; the rising demand for HIV/AIDS treatment and care as Thailand’s epidemic entered the new phase where many of those infected during the early years of the epidemic became ill with symptomatic disease; the changing regional composition of the epidemic; and the broadening scope of the policy response to the epidemic. 6. The report produced new information by investigating whether more could be done cost-effectively, beyond a least-cost approach to meeting the political mandate of 114 APPENDIX H EXAMPLES OF SUCCESSFUL KNOWLEDGE ACTIVITIES making treatment available to all. Accordingly, it considered three additional enhancing scenarios that addressed possible weak points: • Lowering the price of voluntary counseling and testing (VCT) to increase its accessibility and elicit more demand for it • Mainstreaming demand enhancing programs for ART, to help attain and sustain high levels of adherence among patients • Implementing both expanded VCT and augmented adherence, modeled with synergistic benefit between them. 7. The report estimated the number of life-years saved and the cost per each year- life saved for each scenario and, with a reasonable assumption with regard to the value of a life-year saved, concluded that even the combined policy scenario would be attractive. Sensitivity analyses on a key set of assumptions provided further credibility to the calculations. 8. The participation of technical staff from the Ministry of Health helped to build ownership for the study and helped local co-authors in the policy dialogue with the government and in introducing universal coverage for ART. In addition, the report benefited from international experience, including through consultations on costs and the Asian Epidemic Model used in the study. The cost-benefit analysis provided a good foundation for evidence-based policy options, which in addition to those derived from the cost-benefit information on different policy scenarios, covered ways to contain the public financial cost of interventions. C. Development of Recommendations: The South Africa Sector Study of the Effective Tax Burden 9. A study on the cost of red tape for business identified “company and operating taxes” as a significant administrative barrier to business development that also encouraged businesses to evade or elude taxes. In 2005, the government directed attention to the costs and complexity for small businesses of the tax code and the National Treasury requested that the Bank Group’s Foreign Investment Advisory Service (FIAS) conduct a study of the marginal effective tax rate on five key sectors in the South African economy. The study was also part of a multicountry study of African revenue authorities being undertaken by FIAS in collaboration with the U.K. Department for International Development that sought to determine whether the tax policy and tax administration regimes are conducive to economic growth. A key focus was on the opportunities created by bringing informal firms into the tax net and appropriate tax policies for small enterprises. 115 APPENDIX H EXAMPLES OF SUCCESSFUL KNOWLEDGE ACTIVITIES 10. The South Africa advisory project is an example of recommendations developed from findings with due attention to institutional capacity constraints and development. The study concluded that the overall tax system is appropriate, but that it needs to address “second order” general issues and sector-specific issues and differences in effective tax rates. Recommendations focused on these findings and included fairly specific proposals on the broad tax system, such as considering a declining-balance approach to depreciation to improve administration and compliance and moving from first-in, first-out to last-in, first-out accounting to reduce taxation of inflationary gains on inventory capital. Recommendations also addressed specific issues in the five sectors covered, such as reducing ring-fencing in mining (whereby capital allowances can only be claimed against income generated from the mine with which they are associated) to allow mining companies to use all mining projects in their portfolio and allow private equity funds in the financial sector to operate as tax exempt pass-through entities to encourage foreign investment in private equity. Recommendations were developed with an eye on implementation capacity, for example in the declining-balance proposal cited above or the proposed “quick” methods for calculating the value added tax for small businesses. 11. The study reviewed the effectiveness of the institutions based on feedback from enterprises. The study also benefited from a well-developed methodology to calculate marginal effective taxation for different sectors, which allowed for some cross-country comparisons covering manufacturing and services, and where South Africa’s marginal effective tax rate is found to be lower than in a set of OECD countries—India, Kenya, and Zambia—but higher than in Tanzania and Uganda. The government implemented (for example, simplification of turnover tax regime for small business, reduction in tax rates) or was in the process of implementing (for example, financial sector tax issues) many of the recommendations from the study. Furthermore, the study provided the Treasury with an analytical tool for reviewing the impact of tax policy, although feedback from Treasury was that more could have been done to improve its capacity (for example, by greater participation in the study). A follow-up advisory activity focused on tax compliance costs for small businesses and responded in part to the knowledge generated from the marginal effective tax rate advisory. 116 Appendix I Results of World Bank Knowledge Services in the Focus Countries 1. This appendix provides an overview of the World Bank’s engagement in the evaluation’s focus countries and summarizes Independent Evaluation Group’s assessment of the results achieved by the World Bank’s knowledge services in these countries. Bulgaria 2. With increasing access to European Union (EU) funds after EU membership and strong fiscal and external positions before the global economic crisis, Bulgaria’s need for World Bank financing has been diminishing. Up to the FY07-09 Country Partnership Strategy (CPS), the Bank’s partnership with Bulgaria was still designed around a lending program, supported by broad-ranging knowledge activities. This CPS cycle coincided with the initial phase of Bulgaria’s EU membership; hence, the Bank Group support was focused on supporting Bulgaria’s efforts for convergence with the EU. The CPS was complemented by a results framework with detailed outcome indicators and intermediate indicators to track progress, although outcomes were not explicitly linked to planned knowledge activities. 3. With the FY11-13 CPS, the partnership shifted to a knowledge-based program, structured around policy support for Bulgaria’s implementation of the Europe 2020 strategy and support of strategies and institutions for accelerated absorption of EU funds. The new strategy is complemented by a pillar with a limited, but flexible, lending program complementary to EU financing. Based on the memorandum of understanding between the government and the Bank, signed in January 2012, a large part of the Bank’s knowledge products will be delivered as RAS, financed by EU structural funds. This innovative arrangement is to be piloted for the first time by the Bank in Bulgaria but may be extended to other Eastern European EU members as well. 4. The FY11-13 CPS has a detailed results framework that links planned analytical and advisory activities (AAA) to specific outcomes expected to be influenced by the Bank’s program and provides milestones to track implementation progress. However, few measurable indicators to trace progress toward the milestones and CPS outcomes have been elaborated. The Bank and the authorities are planning an annual review of CPS implementation progress. 117 APPENDIX I RESULTS OF WORLD BANK KNOWLEDGE SERVICES IN THE FOCUS COUNTRIES 5. In Bulgaria, knowledge activities were bundled with lending in the past, so it is hard to attribute changes exclusively to knowledge services. The bundling of knowledge activities with development policy loans was a powerful combination, but equally powerful was the incentive to join the EU, which was a strong driver for reforms. The policy dialogue seems to have been very good in both freestanding and lending-bundled knowledge activities. Results have been achieved in several areas, in particular in primary and secondary education, labor market reform, budget management, and the business regulatory environment. Some of the most appreciated contributions were in poverty monitoring and human development policy notes, especially those on labor market and social policies. These World Bank contributions were in key areas that were neglected (poverty monitoring) or required an external review and support for effective policy implementation (labor market and social policies). There was little competition for the Bank’s services in these areas, and that competition was of a caliber not comparable to the Bank’s quality of advice. 6. With support from the Bank’s knowledge services and lending operations results have been achieved in several areas, in particular in primary and secondary education, labor market regulations, budget management, and the business regulatory environment. By contrast, judicial sector reform is an area where the Bank’s advisory support has had negligible impact. Without significant progress on judicial reform, little could be achieved in enforcing new and improved norms and regulations, making it difficult to achieve the goals of improving the competitive environment and the attractiveness of Bulgaria to investment. 7. Results have been partial in some areas, such as education, where progress in vocational and higher education is lagging, while in general education, achievements have been concentrated in financing and school management with education quality and student performance still lagging or even deteriorating in international comparison. Similarly, while the regulatory framework of labor markets has been improved, skills gaps are prevalent and labor force participation remains weak. There is also concern, as documented in the EU Council’s recommendations on Bulgaria’s national reform program, that results achieved in some areas, including with support from the Bank, have not been sustained. Policy setbacks have been observed in particular in tax compliance and collection, in program budgeting, and in the market competition environment. 8. Results on the financing and management of schools are tangible. Optimization of primary and secondary education schools has been completed, with 13 percent of schools closed since 2008, while per capita funding is operational and 118 APPENDIX I RESULTS OF WORLD BANK KNOWLEDGE SERVICES IN THE FOCUS COUNTRIES school budgets have been decentralized across the board. Efforts to improve school accountability are being pursued in parallel, by publishing financial results of schools and setting up school boards with external participation, with support from the Bank. Savings from school optimization and improved financial management have been used to increase teacher salaries (up to two-thirds of savings) and to improve facilities. Many of these measures were supported by the development policy loans, which were underpinned by the Bank’s knowledge activities. But, schooling outcomes remain very low and education reform needs to continue (notably through a revision of the curriculum, better training of teachers) to improve the quality of the education system. 9. By contrast, reforms in vocational training have been slower. Although vocational education is part of the “unified standards” system for per capita financing, the development of curricula has been delayed and skill mismatches in the labor market persist. Higher education remains fragmented, with numerous universities that until recently faced no incentives to merge, reflecting the significant inflow of students. Per capita funding was introduced in 2007, for newly admitted students, and a funding formula for universities is under preparation with technical support from the Bank. A significant part of indicators will be linked to the quality of the curricula and the insertion of students into labor markets upon graduation. 10. Labor market reforms have been implemented, in line with World Bank recommendations. A new labor code was prepared; a voucher system has been introduced to improve the effectiveness of active labor market policies; the payroll tax was reduced in a fiscally sustainable manner, from 32 percent to 29 percent; and the minimum wage increased less than the average wage, thus potentially easing the constraints on formal employment in low-pay industries and micro and small enterprises. A recommendation by the Bank (2007 Country Economic Memorandum) to determine minimum wages sector by sector was not implemented as it was not considered appropriate for the country. A booming economy before the 2008-09 crisis significantly reduced unemployment, but the employment rate, at 60 percent, still remained lower than in comparable EU members. Unemployment increased as a result of the crisis. 11. Results were achieved in poverty monitoring where the Multi-Topic Household Survey was followed up by a narrower series of three surveys to understand the impact on poverty of the global crisis. All this work—carried out with the Open Society Institute—is helping the government improve the design of social programs and understand the distribution of poverty in Bulgaria. On justice reform, although the work was considered of high quality by counterparts, results have been slow in coming because of lack of political will to implement the reforms 119 APPENDIX I RESULTS OF WORLD BANK KNOWLEDGE SERVICES IN THE FOCUS COUNTRIES recommended. The Bank did not follow up on this work, which may have contributed to the weak results. The human development policy notes have had tangible results on labor, social programs, and health sector policies. They were also used as inputs to the government’s national reform platform. 12. The public financial management (PFM) work has achieved mixed results. The Bank provided policy advice on possible options to use optimally EU resources in education, agriculture, roads, and environmental protection as well as technical assistance (TA) to improve performance-based budgeting (PBB) and Public Investment Management (PIM). • In education, a number of policy recommendations were implemented that have contributed to improving efficiency of spending in the primary and secondary education sector, notably a decentralization approach with a new per capita fee-based policy. This resulted in the consolidation of the number of schools and reduction by one-fourth of the number of teachers employed. • In agriculture, most of World Bank recommendations were not followed. Bank advice was to prioritize activities that help to improve competitiveness (Pillar 2 Public Accounts Committee) over income support payments (Pillar 1 PAC). Support schemes that target specific sector adjustment needs, such as farm consolidation and productivity of competitiveness, were considered as the only sustainable solution to agriculture income problems. Instead, the government opted for complementary national direct payment (CNDP) scheme, therefore diverting some resources programmed under the “rural development” (Pillar 2 PAC). Also, in 2009, the government started to introduce specific coupled CNDP (linked to the amount produced), which comes at a high implementation costs and a long-term distorting impact on production patterns, when the Bank had recommended a flat, decoupled CNDP (not linked to the amount produced). Under Pillar 2, the government program lacked selectivity in the measures supported, therefore challenging the national absorption capacity and diluting the potential effectiveness in addressing key rural challenges. Also, the Ministry of Agriculture and Food has not yet developed a National Land Consolidation Strategy to address the fragmentation of land ownership and operatorship issue, which still represents a major impediment to sector productivity. • The important PBB and PIM reform is not yet bearing expected fruit for improved management and allocation of resources. TA led to some improvement in the presentation of budget documents. But in practice, annual budget planning remains largely incremental instead of relying on performance information, which remains poor and not adequately targeted to 120 APPENDIX I RESULTS OF WORLD BANK KNOWLEDGE SERVICES IN THE FOCUS COUNTRIES the need of policy makers and parliament. Responsibilities are fragmented, therefore weakening the link between strategic planning and budgeting and reducing public sector effectiveness and accountability notably for managing public investment. The TA had recommended preparation of a new public finance organic law notably to institutionalize the new budget framework (policies/program/output indicators), to include comprehensive public spending reviews and targeted program evaluation as part of the budget process. But the new public finance law has not yet been submitted to parliament. Performance-related pay was introduced in 2012, but it is too early to assess its impact. 13. Overall, one expected outcome of the TA on PBB and PIM was to restructure/improve budget procedures to enhance the absorption capacity for the EU funds. Bulgaria’s experience over the past five years confirms the experience from other EU15 countries and shows that PFM reforms needs to be complemented by the development of well-defined sectoral strategic programs backed by a sound medium-term fiscal framework to unleash the effectiveness of EU funds. Forthcoming Bank assistance to the Bulgarian government to guide the preparation/finalization of the water, health, and “smart specialization” strategies is therefore most welcome. Knowledge services have helped build support for critical reforms, but it seems to have been more successful when bundled with budget support lending. 14. The Bank’s consistent knowledge work on streamlining administrative procedure have resulted in an improved business environment, but progress has been slowing down, and a more specific concentration on issues like administrative issue at the local and state level may require more TA projects rather than further reports. The knowledge services were of significant importance in the past, and the renewed efforts since the crisis are commendable, but it is doubtful if much more can be accomplished unless attention shifts into specific areas, such as trying to tackle administrative barriers at the local level or engaging in supporting the country’s innovation drive through technoparks. Chile 15. Both the FY2007-10 and the FY2011-16 CPSs were structured around strategic pillars aligned with the government’s program. The pillars were comprehensive enough to encompass several areas of Bank engagement, and the CPSs highlighted the knowledge activities and lending services to be provided under each area. Both CPS documents had results frameworks with outcome indicators specific to each area of engagement. Some of these indicators were elaborate but others were generic 121 APPENDIX I RESULTS OF WORLD BANK KNOWLEDGE SERVICES IN THE FOCUS COUNTRIES (such as, GDP growth above 5 percent) and the attribution of results to the Bank’s activities was difficult. The 2007-10 CPS did not provide a framework to link the expected outcomes to the results of the Bank’s knowledge services. The expected contribution of the Bank’s knowledge services to the reform program was linked by the 2007-10 CPS to the overall usefulness of knowledge services, based on its quality and relevance. The FY2011-16 CPS designed outcome indicators to assess the results of the Bank’s knowledge services. 16. The Joint Studies Program (JSP), cofinanced by the government and the Bank, covers a range of studies that broadly match the strategic priorities of the CPS. The Bank consults with line ministries and other agencies to identify sector priorities, but the selection of the JSP activities is decided jointly with the Director of Budgets of the Ministry of Finance, which has a strong influence on the content of the JSP. The JSP studies typically fall in two categories: • Studies that fill a knowledge gap in an area where the government needs to take action: relevance, timeliness, and actionable recommendations (“how to” as opposed to “what to do”) are key for results. • Studies that explore issues in an area where the government has not taken a position and where the Bank is used as a “sounding board.” Results may not necessarily follow in a short time. 17. Evidence on results is mixed. In some areas the Bank’s knowledge services have had a strong influence on the development agenda—in the financial sector, higher education, and education quality assurance. Following up on the Financial Sector Assessment Program (FSAP) recommendations, with support from the Financial Sector Reform and Strengthening (FIRST) initiative, all listed companies have switched to International Financial Reporting Standards while progress toward risk-based supervision in the non-bank sector has been significant. The enhanced priority that governments assigned to access and quality of education, as a result of student protests in 2006 and 2011, contributed to the government’s interest in working with the Bank and increased the impact of the Bank’s work on quality assurance and student loan policies. Other work in education, including on early childhood development and the secondary school subsidy, had little impact. In some other areas, the Bank’s influence on the medium-term development agenda through the knowledge service instrument was not strong, or has been subject to change because of shifts in government priorities. Examples include the work on regional development where the Bank could have paid more attention to the political economy of incentives at the local level. In innovation policy the Bank’s work was 122 APPENDIX I RESULTS OF WORLD BANK KNOWLEDGE SERVICES IN THE FOCUS COUNTRIES virtually suspended following cancellation of the Innovation and Competitiveness investment loan by the new government in 2010. 18. The Bank generally is seen as responsive and timely, especially compared to other development partners, which are seen as even more driven by a “lending culture” than the Bank. However, in some cases there were significant delays in the delivery of knowledge activities, such as in the work for public sector management improvement and for evaluation systems. In some instances the Bank’s work is perceived as lacking impact because of focus on the “what to do” recommendations as opposed on the “how to” practical suggestions, such as in the TA provided on the management of pension fund financial assets or in the lessons for policy assessment from OECD countries. Work on the decentralization of education and health validated the authorities’ views on the subject, but lacked some practical aspects for implementation. Similarly, some of the work on improving public sector management was conceptually sound but did not refer enough to local Chilean processes to be of practical implementation value. China 19. The Bank’s country program remained firmed tied to lending, with knowledge services supporting, and being supported by, lending services. There was a very structured partnership framework, driven by China’s own development priorities and plans. The results framework presented in the CPS included a vast number of somewhat confusing outcome indicators, but no targets or milestones for monitoring program progress. It was inadequate to assess the impact of the Bank’s knowledge or lending services. 20. The Bank’s knowledge service program was jointly selected by the Ministry of Finance and the Bank following an annual review and budget allocation process. With the exception of FY10-11, it was a well-managed process that ensured a focus on strategic priorities. The Bank’s knowledge program became a combination of big, formal studies and short, informal notes, produced on request to inform particular clients. There was unmet demand for Bank’s knowledge services and most, if not all, focused on the CPS strategic priorities. Selectivity, therefore, had been exercised by both the Ministry of Finance and the Bank. There was also considerable flexibility in the design and delivery of knowledge services through a wide variety of collaboration modes between the Bank and the counterparts. 21. The results were mixed for the knowledge services delivered in China between FY05-11. In private sector development, the reviewed knowledge services focused on investment climate, innovation, and competitiveness issues and 123 APPENDIX I RESULTS OF WORLD BANK KNOWLEDGE SERVICES IN THE FOCUS COUNTRIES provided policy recommendations based on international examples and firm-level surveys. Overall, the clients felt that while most studies were of high quality, their influence was limited by weak dissemination and lack of follow-up support. 22. In the financial sector, the Bank’s role has become minor given the size and complexity of financial markets in China. Among the dozen of economic and sector work (ESW) and technical assistance (TA) products reviewed, only three (the FSAP, the Capital Market Report, and the Report on the Observance of Standards and Codes [ROSC] on accounting and auditing) have had measurable traction, prompting the authorities to take action to address the weaknesses in the financial system and to improve the accounting standards. Dissemination and impact of other papers were limited. 23. In human development sectors, the Bank’s knowledge services have been effective in conveying reform messages, highlighting options and generating discussions. In many instances, the Bank’s work has influenced the reform process. This has extensively been the case in social assistance and to a certain extent in the pension system. Both areas are still evolving and interactions with the Bank continue. 24. In infrastructure and environment, practically all knowledge activities were concurrent with lending even if the activities and the lending did not always focus on the same topics. Such blending facilitated dialogue and helped knowledge activities’ effectiveness. Some tangible results include the establishment of provincial energy conservation targets, the creation of the National Energy Bureau, the adoption of low carbon measures at the city level, the policy changes in the use of fuel tax revenues, and others. There is also some progress in the railway sector, where the Bank’s work has led to internal debates on railway multioperators and the multiplication of joint venture operators over the past six years. 25. In rural development, although there is still a long way to go, there is evidence that the World Bank’s knowledge services have been effective in achieving the outcomes sought in the CPS, such as in the economic value of water, a topic of fundamental importance to the sustainability of China’s transformation, not only in the North China Plain but countrywide. 26. In addition, the World Bank Institute’s technical assistance support to the International Poverty Reduction Center in China, a key platform for sharing China’s development experience, has contributed significantly to strengthening the center’s capacity to conduct training and research, and has an important role in improving China’s aid quality. 124 APPENDIX I RESULTS OF WORLD BANK KNOWLEDGE SERVICES IN THE FOCUS COUNTRIES 27. There is evidence that the Bank’s knowledge services have contributed to strengthening the analytical and policy formulation capacity of the clients. The contribution has been the strongest when client capacity building was a clearly stated objective of the knowledge activities (for example, work with the International Poverty Reduction Center), or there was very close collaboration between the Bank and the client (such as the work with Ministry of Science and Technology), and the weakest when the client or counterpart was mainly relied on for data collection (as with the National Bureau of Statistics, China Academy of Social Science). There is demand for greater effort on the part of the Bank to involve the counterparts more in analysis and the formulation of recommendations. This would enhance ownership and interest in following up to produce results. Kazakhstan 28. The World Bank’s knowledge-based country program with Kazakhstan is anchored by the Joint Economic Research Program (JERP). 30 The JERP is a demand- driven cofinanced program whereby the government and the Bank annually agree on the knowledge services program. This ranges notably from policy analysis to strategic planning to good practice notes to brainstorming. The JERP, now entering its fourth three-year cycle, has been growing over time, notably during the 2008-09 crisis, from $1.3 million in 2004 (of which 40 percent was government financed) to $4.3 million in 2012 (of which 85 percent was government financed). The JERP has definitely contributed to increasing the relevance of the Bank’s work in Kazakhstan. 29. The 2004 strategy document did not include a results-based matrix, as this was not mandatory in 2004. The 2012 CPS presents a results framework over FY12- 17 presenting major milestones and outputs, and expected outcomes. The matrix lists proposed JERP activities in support of each of these outcomes. 30. The JERP is fully demand-driven, ensuring strong government ownership of the knowledge services program. The government asks all agencies to make a request for proposal for advisory services. The Ministry of Economy then reviews the proposals with the Bank team through several rounds of discussions before finalizing the annual program. Overall, the Bank concentrated on areas covering a coherent critical mass of reforms (for example, public finances and public sector). Knowledge activities in the private sector focused on diversification, deregulation, and competitiveness, which reflect the Bank’s strategy for Kazakhstan’s private sector. Given the development of the education sector, the focus of the Bank’s work on secondary and tertiary education was warranted and provided sufficient selectivity. 125 APPENDIX I RESULTS OF WORLD BANK KNOWLEDGE SERVICES IN THE FOCUS COUNTRIES 31. The flexibility of this program allowed it to adapt to changing government priorities/country context and has contributed to its growth since 2004. For example, the Bank responded well when asked for advice during the financial crisis. Brainstorming sessions upon a request from the Prime Minister to provide quick response on a large array of topics to inform policy decisions. In addition, the work on poverty monitoring included an analysis of the potential implications of the global economic downturn for poverty in Kazakhstan. However, there were areas where the program could have benefitted from a stronger strategic focus, notably governance. 31 32. The new 2012 CPS has acknowledged this challenge and plans to transform the JERP into a programmatic structure. The objective of this new approach would be to strengthen the strategic focus of the program to address medium-term challenges through improved task sequencing, to improve the coordination of tasks emphasizing interconnected pieces of work to better understand and address policy linkages, and to better track impact. 33. The pace of reform implementation often depends on both the drive of ministries and the overall external environment, in part linked to oil price fluctuations. But overall, the highest impact came from high-level brainstorming sessions because the Prime Minister requested and chaired them, had a strong interest in the topics, and ensured that the right officials were present. These sessions were particularly helpful in developing the authorities’ ability to systematically think through problems with substantial analytical support from the Bank. The impact of the sessions is evident in several areas: resuming pre-school education services, providing input for the elaboration of the anti-crisis package, establishing per capita financing in health system, and providing impetus to a program on urbanization. Brainstorming has become a method for debating important policy issues in Kazakhstan, and the Bank has become an integral part of the policy process. By contrast, evidence on results achieved through other engagements under the JERP is mixed across sectors. 34. Results in public sector and economic policy work are diverse. In oil management, the government adopted the concept of national fund development in 2005 and after the crisis, in 2010, issued a presidential decree to cap annual transfers from the National Fund to the budget. However, until recently the rule did not embed a countercyclical approach into fiscal management as per Bank advice. 32 The macro-modeling project with the Ministry of Economy has strengthened the recipient’s capacity to prepare macro forecasts and policy simulations. On tax policy and administration, the Bank delivered high-quality advice that led to the adoption 126 APPENDIX I RESULTS OF WORLD BANK KNOWLEDGE SERVICES IN THE FOCUS COUNTRIES of the 2008 tax code. The Bank also provided analytical support in revenue administration, which was followed by two projects to assist in the implementation of modernization strategy for tax and custom administration. However, results have been much slower to materialize than in tax policy. In sovereign debt management, although a new debt management division has been put in place at the Ministry of Finance, other recommendations, such as developing a sovereign debt management strategy and deepening the domestic secondary market for sovereign debt (still virtually non-existent), remain pending. A new unit created in 2011 within the Ministry of Economy formulates government policy with respect to management of public sector financial liabilities. Also, JERP TA provided by the Bank in FY12 helped develop a medium-term debt management strategy expected to be made public this year. In public administration reform, Bank advice contributed to the 2008 revision of the budget code and the introduction of a system of strategic planning in 2009. Nevertheless, much remains to be done to improve the overall quality of sector strategies and merge strategic planning with budget preparation. The government has also somewhat strengthened institutional and management capacity in procurement, along the lines recommended by the Bank. On intergovernmental fiscal relations, the government adopted a new formula on transfers to equalize provision of basic education and health services across the country. But the broader agenda of decentralization has not gone very far and is up against a strong culture of centralized decisions. On the World Trade Organization (WTO), the Bank’s advice was targeted and well-taken by the government, but the timing of accession has been delayed several times, initially because of regional trade interests that conflicted with WTO requirements, and more recently attributed to the intricacies of how national policies and interests would be affected by WTO accession. 35. Results are also diverse in the financial sector. Following the financial crisis, after the organization of two brainstorming sessions at short notice in 2008 and 2009, the government set up a special commission to tackle the banking crisis and used inputs from these discussions to prepare an anti-crisis program. On accounting and audit standards, the Bank’s ROSC provided the framework for the implementation of International Financial Reporting Standards and International Accounting Standards in Kazakhstan, which has been achieved for all large companies and financial institutions. The Bank provided useful input into identifying the key issues to be addressed on insolvency issues. The first phase of the new legal framework has been presented to parliament. Work is ongoing on further changes to the law as well as enhancing the regulatory and insolvency regime. The FSAP update delivered a series of recommendations, which focused on evident vulnerabilities in the financial sector. Overall, the authorities in Kazakhstan, especially the National Bank, took 127 APPENDIX I RESULTS OF WORLD BANK KNOWLEDGE SERVICES IN THE FOCUS COUNTRIES action in relation to the key issues identified in the FSAP Update, notably: putting in place new measures on risk management and internal control, strengthening the provisioning standards on loans (particularly those on off shore borrowers and other risky categories), tightening bank capital adequacy requirements of banks, and introducing new instruments for stress testing in relation to liquidity and credit. The authorities also took specific measures in relation to the resolution of problem banks and problem bank assets. The quality of the work delivered by the Bank on financial systems enhancement was good and again the authorities took action on the results. The major issues highlighted were the heavy reliance of the system on external finance and the increasing level of credit risk associated with the rapid growth in credit due to strong economic growth. The authorities followed up on this by tightening some of the liquidity and capital requirements especially targeted at external finance with the aim of making this a less attractive source of finance. In addition, the authorities reviewed and strengthened the classification and provisioning standards for certain categories of loans as well as placing limitations on the growth that could take place at some second-tier institutions. 36. World Bank knowledge services concentrated in three areas in private sector development: diversification of the economy, competition policies, and removal of regulatory restrictions. The Bank conducted work on economic diversification by examining options to establish supply chain development in Kazakhstan’s lopsided oil, gas, and minerals economy. While the study built on the way other oil rich countries had successfully diversified and suggested possibilities for Kazakhstan to follow suit, no major step toward diversification has been taken since the study was delivered in 2008. Other knowledge activities also studied the possibility of establishing new, high-tech companies. The analysis of the existing technoparks formulated a critical assessment in 2008-09, but no further efforts have been undertaken to tackle this issue. Half a dozen TA projects on competition policies focused on the major issues arising from the strong industrial concentration within the country. Several World Bank teams reviewed the competition law and policies, while the European Bank for Reconstruction and Development studied the competitive aspects of such subsectors as telecom and electricity. Subsequent works analyzed issues of financial sector market power and consumer protection. Major accomplishments were contributions to the Law on Protection of Consumers’ Rights, including work on corporate governance and the role of credit bureaus. In the area of regulatory restrictions, the authorities had extensive consultations with Bank staff with regard to the findings of the Kazakhstan Doing Business report. The overall acceptance of the original low standing led to efforts to remove regulatory barriers to businesses. Those efforts have paid off as, according to Doing Business, the 128 APPENDIX I RESULTS OF WORLD BANK KNOWLEDGE SERVICES IN THE FOCUS COUNTRIES country has become by far the least regulated economy in Central Asia and the Russian Federation, reaching below 50 in the rankings for the first time in 2012. 37. In higher education, knowledge activities did not lead to significant policy reforms. Although ministry official reported closing ineffective institutions as a result of the Bank/OECD report, the recommendation was rather to establish a quality assurance system and through this system rationalize weaker institutions. This advice was not implemented. The Bank’s advice appears to have been useful notably in establishing the new university. In secondary education, Bank TA spurred positive changes in assessment methodology, notably the inclusion of items to measure higher-level cognitive skills. However, the need for further changes has re-emerged after the publication of PISA results. 38. On social protection, the Bank’s knowledge activities have contributed to building technical capacity (such as, Pension Reform Options Simulation Toolkit training for pensions) and led to policy changes. The introduction of a basic pension and consolidation of pension accounts were done as result of Bank recommendations. Moreover, in order to improve the social protection system, the government is thinking to implement a conditional cash transfer program with the assistance of the Bank. Through programmatic Bank assistance under the JERP the government is in the process of phasing in a conditional cash transfer program, complementing labor activation measures through cash incentives to use employment, health care, education, and other social services in a comprehensive manner. Kuwait 39. The Bank’s knowledge-based country program provides for a structured partnership framework, based on a formal CPS document, with focus on strategic priorities. However, there was no result framework even in the 2010 CPS. All Bank knowledge services are funded through RAS. The Bank focus was on the strategic priorities stated in the CPS but remained flexible to respond to emerging needs, such as the governance advisory in response to the financial crisis and the land transport advisory. A significant and complementary volume of advisory services was provided on PFM issues. Diagnostics and recommendations were mutually reinforcing (for example, procurement, public expenditure and financial accountability, civil service reform, and governance and integrity program). However, in the private sector, the program was more ad hoc. The financial sector program consisted of the joint FSAP and the bank governance project. 129 APPENDIX I RESULTS OF WORLD BANK KNOWLEDGE SERVICES IN THE FOCUS COUNTRIES 40. Most activities were conducted by the Bank but with significant government participation. Indeed, the government set up committees to work with the Bank team to provide information and data and participate in the analysis (Public Expenditure and Financial Accountability[PEFA], Country Procurement Assessment Review [CPAR], and FSAP) and dissemination. Ownership was strong as activities were demand-driven based on specific needs of each organization. This strong participation facilitated the acquisition of country knowledge. However, the depth and breadth varies across sectors. In public finance, these activities provide comprehensive coverage of country knowledge (public finance management, procurement, and civil service reform issues) while in the private sector, activities were more narrowly focused. 41. Overall, expected results have not yet materialized for most projects. One reason is that many activities called first for a new law before reform could be implemented, and interactions between parliament and the executive complicated the passing of several key laws (procurement, public finance, civil service, freedom of information, anticorruption). In some instances, the government has started engaging or piloting some recommendations (formulating a new budget classification in PEFA, piloting a new job classification and salary scale in civil service reform). Nevertheless, recommendations remain high on the political agenda. Most reports provided ammunition for counterparts to continue to pursue these reforms using the evidence provided in the reports in internal discussions within the council of ministers and/or the parliament (review of spending allocation report, procurement, anticorruption, banking governance). 42. In education, the government requested Bank support to address quality and inefficiency in the sector, with the aim of promoting economic diversification. The TA program comprised, as most prominent outputs, a Public Expenditure Review and the Kuwait Education Indicators and Assessment Program (KEIAP). The program was designed to inform new laws and regulations, reform government programs, build recipient capacity to improve information systems, and indirectly improve the school as an institution. The education expenditure analysis study was conducted jointly with government counterparts. For the KEIAP, the World Bank operated in a true partnership with the Ministry of Education and the three teams designated as counterparts, with the objective to build capacity. However, political ownership was lost from 2006 to 2009. As a result, KEIAP was conducted jointly only at a technical level. 43. Overall, little use was made of the outputs. Three outcomes were fully achieved: producing education indicators according to international standards, 130 APPENDIX I RESULTS OF WORLD BANK KNOWLEDGE SERVICES IN THE FOCUS COUNTRIES participating in the international assessment programs, and establishing a new education management information system which is operational. Two other outcomes were partially achieved: One national learning assessment was realized but no permanent system has been established and a capacity in data production and data analysis has been created in the Ministry of Education, but key members of the team in the planning department have transferred to the training and development department. Using data for planning and promoting policy reforms was not achieved. Malaysia 44. There has been no CPS since 1999. However, the Bank entered into a Framework Agreement with the government in November 2009. The agreement defines respective legal responsibilities of both parties for advisory services. No pillars or specific areas of work were specified in the agreement. However, it was implicitly understood that the focus of the partnership would be on the three goals of the New Economic Model of Malaysia: high-income, inclusiveness, and sustainability. 45. Malaysia is most interested in the independent views of the Bank. Before the Framework Agreement was signed, the Bank’s activities in Malaysia centered on the competitiveness agenda. Two ICAs were prepared as well as a study on improving the measurement and performance of the services sector and a study on the knowledge economy. These issues continue to be relevant to the country’s development challenges. Under the Framework Agreement, requests for fee-based services are coordinated by the Economic Planning Unit of the Prime Minister’s Department. These requests reflected key development constraints facing the country. Often these issues have been debated internally for some time or have already been included in national development plans. The requests therefore reflected the client’s desire for the Bank to provide the intellectual underpinning (theoretical and empirical evidence) for policy measures under consideration. All studies addressed key aspects of building a high-income economy—raising productivity, increasing industry value added, improving equality, improving corporate governance, and raising standards. 46. In addition, the Bank provided fully funded knowledge services, such as the Malaysia Economic Monitor series and ROSCs (corporate governance and accounting and auditing). Each Economic Monitor includes a macroeconomic update and an analysis of a topical issue. For these, the Bank selects the theme for each publication in consultation with the Economic Planning Unit, academics, local think tanks and other relevant parties. The Economic Planning Unit organizes the 131 APPENDIX I RESULTS OF WORLD BANK KNOWLEDGE SERVICES IN THE FOCUS COUNTRIES launch for each Economic Monitor. ROSCs were prepared in close collaboration with the government agencies in charge (Securities Commission and Bank Negara). 47. The results for the knowledge services delivered in Malaysia between FY05- 11 were generally positive, with some gaps in implementation. For some studies, results consisted of inputs into the 9th or 10th Malaysia Economic Plan, and sectoral strategic plans (for example, the National Higher Education Strategic Plan 2007- 2020). Examples of this included the Knowledge Economy study (FY08) and the ICA (FY09). The study on Moving up the Value Chain (FY11) led to a shift in using value added instead of foreign direct investment as a key indicator of industry performance. It supported and accelerated reforms in solar industry that were ongoing, and gave a push to the Medical Device Bill, which was already in the pipeline. It also led to inclusion of medical devices as an Entry Point Project in the Economic Transformation Program. However, implementation of the study recommendations is lacking. The study on Optimal Minimum Wage (FY12) was commissioned when the government had already drafted legislation, but there was significant difference of opinion among the different parties. The Bank’s study provided the analytical framework and cross-country comparison to enable the government to move forward with the enactment of minimum wage legislation and establishment of National Wage Council. Corporate governance ROSC, accounting, and auditing ROSCs were used by the Securities Commission to shape relevant reforms, some of which required legislative changes. Russian Federation 48. The Russian Federation’s fast growth and strengthening of public finance position before the 2008-09 global financial crisis provided for the accumulation of important financial buffers and led to a change in the partnership with the World Bank as financing needs diminished. The CPS for FY2007-11 outlined a transition to a business model mainly revolving around knowledge services, with in parallel a stronger focus of World Bank Group activities (analytical work and investments) on select regions. The FY2007-11 CPS was designed around four pillars, well aligned with the government’s own strategy: sustaining rapid growth, improving public sector management and performance, improving the delivery of social services, and enhancing the Russian Federation’s global role. 49. The 2007-11 CPS had a detailed results framework that linked Bank Group activities, including knowledge services, to specific outcomes expected to be influenced by the program and provided milestones to track implementation progress. However, few measurable indicators were elaborated to objectively trace progress toward the milestones and outcomes of the CPS. In some cases, specific 132 APPENDIX I RESULTS OF WORLD BANK KNOWLEDGE SERVICES IN THE FOCUS COUNTRIES targets were assigned to the CPS milestones, but without direct links to the Bank’s activities. Examples include the target of preventing real appreciation of the ruble in excess of 7 percent, or of keeping federal expenditures below 18 percent of GDP. 50. The FY2012-16 CPS further positions the Bank Group as a partner of choice for the Russian Federation in the provision of knowledge services. The CPS is designed around four pillars, in continuation of the themes of the previous CPS: increasing growth and economic diversification, expanding human capital by strengthening skills and social services, deepening the Russian Federation’s global and regional role, improving governance and transparency (cross-cutting CPS theme). The CPS emphasizes the flexibility and speed in the provision of knowledge services, with particular focus on reimbursable TA, to respond to evolving demands of clients. It also continues to emphasize strengthening Bank Group activities in the Russian Federation’s poorer regions. The results framework of the 2012-16 CPS includes 14 outcomes and milestones. As in the FY2007-11 CPS, there are only a few measurable indicators to trace progress toward the outcomes of the CPS. 51. Over FY05-11 the Bank spent $27.6 million to deliver 120 knowledge products, with a steep upward trend in recent years as a result of the initiation of RAS in FY08. The total knowledge service budget over FY08-11 reached $19.4 million, funded up to 46 percent by RAS, with 74 AAA projects delivered, of which 35 were under RAS. The vast majority of RAS, covering 33 knowledge products and 68 percent of the RAS budget over FY08-11, were delivered to regional governments and to other non-federal institutions. 52. The growing demand for RAS by regions indicates high relevance of the Bank’s global knowledge for development solutions at the subnational level. At the same time, demand for RAS comes mostly from the Russian Federation’s wealthiest regions. The Bank is exploring innovative solutions to direct reimbursable TA to the poorer regions. Options under consideration include the creation of a federal fund with participation of the International Bank for Reconstruction and Development to provide subsidized reimbursable TA to less advanced regions. 53. At the same time, the considerable increase in knowledge activities conducted under RAS since FY08 has reduced the predictability of the knowledge service program. Among the 19 originally planned ESW projects in the FY07-11 CPS, 9 were dropped and 18 new tasks were added. Among the 14 planned TA tasks, 4 were dropped and 62 new tasks were added. A program largely funded through fee- based services requires more versatility in the selection of tasks to ensure its relevance to changing client demands, but this added flexibility will need to be balanced against the risk of fragmentation and loss of strategic focus as a result of 133 APPENDIX I RESULTS OF WORLD BANK KNOWLEDGE SERVICES IN THE FOCUS COUNTRIES the drive to respond and accommodate multiple previously unforeseen needs. To partly respond to these challenges, the Bank has set up an implementation and results monitoring system that would be used to track progress toward achieving the outcomes in the results framework of the 2012-16 CPS. Based on the results from the monitoring, the team would also conduct an annual CPS implementation review for assessing progress and possible adjustments to the World Bank Group program. The Russian Federation 2012-16 CPS is one of the first in the Bank to use the RIMSys (Results Integration and Management System) tool developed by Operations Policy and Country Services (OPCS) to monitor CPS results and progress. RIMSys allows the online monitoring of the implementation progress of the CPS outcomes and milestones with linkages to the various Bank activities. 54. Evidence on results achieved is mixed across sectors. For much of the World Bank knowledge services in private sector development and some of the IFC Advisory Services (mainly on administrative barriers and land issues), the results have mainly been in raising awareness on competitiveness issues, innovation, and the cost of doing business. However, outcomes and impacts have been minimal for two main reasons: first, because transforming the general recommendations into actual policies and programs has been a difficult task for policy makers, and second, because many of the knowledge activity recommendations do not take into account regional differences, which remain important in the opinion of policy makers. Moreover, often World Bank knowledge work consists of studies and short workshops, with no systematic support for policy or program design and implementation as well as institutions and skills building. The results are mixed in the financial sector knowledge products. The FSAP Update and the Post FSAP TA in 2009 in relation to the Payments System were the best examples of TA achieving results. 55. There is some evidence of effectiveness in public sector reforms (covered by the civil service TA advisories); in human development reforms (such as, improving payment systems to human development providers); and in labor markets (such as, better monitoring of employment programs); but not from the poverty work, which did not have policy recommendations and was poorly disseminated. Results of ESW on regional development so far have been minimal, as the regional strategy has yet to be completed while obstacles to the movement of labor and capital are still prevalent. Some changes that facilitate capital mobility have been made in land regulation, but significant impediments to capital mobility remain due to differences in business regulations across regions. The very imperfect housing market remains a major impediment to the mobility of labor, while the registration system of workers remains in place and is a source of corruption. 134 APPENDIX I RESULTS OF WORLD BANK KNOWLEDGE SERVICES IN THE FOCUS COUNTRIES 56. Overall, the work on the PFM agenda has been fruitful and was able to achieve significant results through sustained cooperation over many years. Bank assistance was most helpful in three areas: fiscal federalism where the Bank was able to customize assistance to help the move toward decentralization with the design of a system with new procedures and transparency mechanisms; treasury reforms with the introduction of principles to budget execution and the modernization of treasury functions; and performance-based budgeting where the Bank helped to shape the overall reform at the federal level and selectively pilot performance management and performance budgeting at the subregional level. 57. In infrastructure the strongest results have been in the introduction of public- private partnerships to attract private investments to sectors such as transport, and in energy efficiency. The knowledge activity on the concession via public-private partnership of facilities in Pulkovo airport, St. Petersburg, was highly successful, as it led to a contract with a group of investors and IFC lending for improving and managing those facilities. This partnership, which involves investments of about €1.2 billion, received several awards, including Deal of the Year 2011 by the Infrastructure Journal. In energy efficiency, a federal law on energy conservation approved in 2009 was influenced by a Bank knowledge activity, and a federal program launched in 2010 used tables from the knowledge activity report. In housing, the results appear more tenuous, more a contribution on assessment methodology whose impact appears to be less significant than originally expected. 58. In agriculture and rural development, there is evidence that the support for improving the processes and management of the public expenditure program for agriculture has been quite successful, though expenditure priorities remain problematic. Nonetheless, performance-based budgeting was adopted in full, with the Ministry of Agriculture serving as a successful pilot. The outcomes of knowledge activities related to improving efficiency in grain marketing arrangements, especially for exports, were disappointing. In general, the Bank’s advice was disregarded when a state grain trading enterprise was established. Bank advice was again not accepted when the Russian Federation embargoed grain exports during the 2010 international grain crisis. By contrast, the Water Quality Monitoring TA was an unqualified success: techniques were tested in the Iset River Basin in parallel with preparation of a major revision to the Federal Water Law. The knowledge generated from the Iset River Basin was subsequently transferred to assessments and mitigation efforts in 10 other river basins, including the Amur, Don, Koma, Neva, Ob, Ural, and Volga Basins. 135 APPENDIX I RESULTS OF WORLD BANK KNOWLEDGE SERVICES IN THE FOCUS COUNTRIES South Africa 59. The CPS was designed around two pillars: urban and rural development and regional integration, underpinned by a cross-cutting theme on capacity building for public service delivery. The CPS provides for outcome and indicators to assess the results of Bank knowledge services. However, most of them are statements of intent without specific quantitative indicators against a baseline. Overall, counterparts are interested in knowledge that would help build capacity either through training, networking, or access to international best practices. In some areas, there is high sensitivity to embracing Bank policy recommendations due to the political economy in South Africa, especially in sectors involving labor issues. There has been only one reimbursable TA program with the Department of Rural Development and Land Reform. For this land reform program, the Bank’s policy recommendations generally were given serious consideration. 60. Knowledge products showed mixed results. In macro modeling, the Bank made a strong contribution in developing computable general equilibrium modeling capacity within the Treasury. However, the engagement in this area has been terminated as a new counterpart team felt that the policy simulations were largely driven by the Bank’s own research and policy agenda. Nevertheless, Bank work was taken into consideration as an input in designing policies such as the wage subsidy scheme and the carbon emission tax, but both schemes are yet to be implemented. 61. In private sector development, overall knowledge work led to the implementation of selected measures, such as reforming turnover tax requirements for small and medium enterprises (SME), establishing a credit bureau, merging agencies involved in SME financing/guarantee, and strengthening credit bureaus. The Endeavor Entrepreneur Institute continues to support high impact entrepreneurship and is one of the leading business support provider to SME. Conversely, support to the Department of Trade and Industry has not produced expected results, due to disagreements on investment promotion and competition policies. IFC support to entrepreneurship programs was mainly financial, with little knowledge transfer; for example, the Women Entrepreneurial Program was not sustainable once the funding ceased. 62. In the financial sector there has been a meeting of minds between the World Bank and the authorities, leading to results. One example of this is the introduction of a new regulatory regime (the so called twin peak program), which would separate prudential and operational regulations with the South African Reserve Bank being in charge of prudential regulations. Another example is strengthened 136 APPENDIX I RESULTS OF WORLD BANK KNOWLEDGE SERVICES IN THE FOCUS COUNTRIES coordination among regulators and clearer guidelines on bank resolution. However, an area where the progress has been slow is the creation of a deposit insurance fund. 63. The Bank’s analytical work on land reform was useful and contributed to broad-based discussions on the subject. For example, exposure of program administrators to international experience through study tours abroad (China) and in-country training courses, has increased the knowledge of counterparts and enabled a joint effort to formulate program refinements and address implementation issues. Also, the Bank contributed to survey-based rural sector profiling to facilitate scaling up of this critical database to encompass the 160 wards the government plans to cover by 2014. However, the land delivery to black farmers under the program was 7 percent in 2011, well below the national target of 30 percent by 2012 stated in the CPS. However, there is a far greater volume of land transfers between white and black farmers occurring outside of the official program, but nonetheless stimulated as a “willing buyer/willing seller” spillover. Surveys of deeded transfers conducted by faculty and staff at the University of Pretoria show that the actual magnitude of transfers may already have exceeded the 30 percent target. Thailand 64. Both the FY03-05 CAS, which was carried forward through FY09, and the 2011 strategy document were structured around strategic pillars aligned with the government’s program. Although the CAS envisaged the possibility of additional lending, the emphasis was on facilitating knowledge sharing to achieve CAS objectives. The centerpiece of this knowledge relationship was the Country Development Partnerships (CDP), supported by comprehensive diagnostic overviews at general economic and sectoral levels (for example, the Bank’s Economic Monitor series and periodic policy reviews). The strategy document reiterated the central role of AAAs in the Bank’s Thailand program, and proposed to use fee-based advisory services and reimbursable TA to support the CDPs. 65. There was at least one CDP for each of the five pillars of the CAS. A rather complicated results framework, with measurement, partnership, implementation, and risk elements, was presented in the CAS and intended to be used for each CDP. The Bank’s Monitors and other diagnostics did not have a results framework, but were to serve as additional monitoring tools to track the outcomes and impacts of the CDPs. The strategy document provided a results framework at the country program level, with outcome targets to measure results. 66. While the CDP arrangements may not have been strategic with regard to individual interventions, they were set in a strategic context. Thus, the poverty CDP 137 APPENDIX I RESULTS OF WORLD BANK KNOWLEDGE SERVICES IN THE FOCUS COUNTRIES addressed issues in poverty analysis and monitoring; the Social Protection CDP in labor markets, safety nets, social insurance, and data collection and dissemination; and health CDP in HIV/AIDS treatment and prevention, universal coverage, and the evaluation and dissemination of the Thai experience. The 2010 and 2011Human Development CDPs focused on elements of public sector reform and health human resources as well as being the conduit for a public financial management report on the health sector and a skills report in the education sector. 67. Although counterparts expressed appreciation of the relevance and quality of the Bank’s AAA, evidence on measurable results achieved is limited and varies across sectors. Client ownership appears to be rather weak for most knowledge activities reviewed in private and financial sector development. The Bank’s work was appreciated, sometimes used as inputs for strategies and reform proposals, but there is very little evidence that the client really tried to follow up on it. A notable exception was the Forum, which the Ministry of Finance organized jointly with the Bank and made a strong effort to pilot and introduce new initiatives (such as a rice insurance scheme, public-private partnership, and public debt modeling). Nevertheless, it is difficult to conclude that such knowledge product and delivery practice ensures stronger client ownership and higher impact. However, it does underscore the importance of providing follow-up implementation support and training to clients in order to strengthen their ownership and to achieve results. The two ICAs conducted were appreciated but, as concluded by the second ICA, there was little improvement in the investment climate in Thailand. Private sector representatives believed that such studies, despite their technical quality, would not produce concrete results without a follow-up implementation strategy. The report Toward a Knowledge Economy in Thailand provided inputs to the National Plan, but there is no evidence that any of the recommendations have been implemented. 68. The FSAP produced many tangible results. For example, a new regulator for the insurance industry, the Office of Insurance Commission (OIC) was one of FSAP’s recommendations. Risk-based supervision was also adopted by the OIC. Supervision of commercial banks has been improved. The Bank of Thailand was reformed to increase its autonomy and the governor is appointed by the parliament. Similarly, most recommendations of the ROSC on Accounting and Auditing have been accepted by the key stakeholders. Implementation is being carried out in stages. 69. In health, The Economics of Effective AIDS Treatment report has been useful in providing the needed cost-benefit information to the government to increase the financing of ART for AIDS patients; and the Harm Reduction Policies and Interventions 138 APPENDIX I RESULTS OF WORLD BANK KNOWLEDGE SERVICES IN THE FOCUS COUNTRIES for Injection Drug Users report contains recommendations that are being implemented by nongovernmental organizations in their HIV harm reduction programs. The Bank has had a broad involvement in knowledge activities ranging from universal coverage to hospital autonomy to human resources, poverty impact of health insurance coverage, and public expenditure. However, the marginal value of additional Bank knowledge in health is diminishing, as the ministry and the health insurance organization have strong analytical capacity and are being supported by European universities. The recent skills report is seen as too general in its recommendations, but the Ministry of Education recognizes skills building as a top reform priority. None of the Bank’s education reports were mentioned as being influential in promoting local thinking about solutions. Still, the government is using thematic studies produced by the health and education teams of the Bank, including from the Development Economics Vice Presidency and network management, and the government is most interested in the experience of other countries. 70. In governance and public sector reform, the Bank’s knowledge service program has been broad and generally of high quality. The very broad scope of the program entails a risk of spreading resources thin in a fragmented manner, to the detriment of the sustainability of the engagement and the achievement of results. There is only partial progress in the key recommendations formulated by the 2009 Public Expenditure and Financial Assessment. The unification of the chart of accounts is in process but both budget preparation and execution are still on a cash basis as the implementation of accrual budgeting has proved difficult. There is still difficulty in providing a reliable consolidated financial report for the general government as financial reporting by local administrations is weak. Progress has been achieved on budget auditing as financial statements for the central government budget up to 2010 have been submitted to the auditors of the budget. The Public Financial Management Report conducted a survey to collect information on spending by local administrations, filling a major information gap in general government financial reporting. No visible change has yet been observed in rebalancing expenditures to the regions or in changing the local-central government arrangements for public service provision. The report has the potential to contribute to these reforms with adequate dissemination and engagement. 71. The Tax Policy review was used as an input to the government’s 2011 tax reform, although only a limited number of recommendations have been implemented. In line with the recommendations, the corporate income tax rate was reduced from 25 to 23 percent, but there was no meaningful revision of holidays from the tax. The TA on tax incidence was primarily used to inform the government’s internal discussions on tax reform, but the Tax Revenue Department is still not able to properly use the model developed. As a result of the TA on 139 APPENDIX I RESULTS OF WORLD BANK KNOWLEDGE SERVICES IN THE FOCUS COUNTRIES transfer pricing, the Tax Revenue Department has a deeper understanding of OECD experience and how it can be applied to Thailand. Similarly, the TA on risk-based tax audits has helped the Tax Revenue Department to develop new criteria for taxpayer classification that have already produced results in shortening the time it takes for big exporters to get a value added tax refund. The TA on procurement contributed to better understanding of the scope of e-auctions, and to the process of establishing a centralized procurement unit. In anticorruption policy, with support from the Bank, the Asset and Liability Declaration forms for public officials were redesigned, with the aim of using them more as a prevention tool than as a cumbersome verification process, so as to be able to conduct more targeted verifications. 140 Appendix J People Interviewed Government Officials in Partner Countries BULGARIA Mr. Assen Antov, former Deputy Head of Road Infrastructure Agency Ms. Snejana Blagoeva, Director, Direct Payments Mr. Lyubomir Datzov, former Deputy Minister of Finance Mr. Mihail Dragiev, former Chief Financial Officer of Bulgarian State Railways(BDZ EAD), currently Chief, Elprom Ms. Iva Georgiav, State Expert, Foreign Relations, State Energy, and Water Regulatory Commission (SEWRC) Ms. Miroslava Georgieva, former Research Director, Ministry of Agriculture and Food Ms. Mila Georgieva, Office of the Prosecutor General Ms. Slavka Kamenova, Supreme Judicial Council Ms. Dobrinka Krasteva, former Minister of Finance Mr. Lazar Lazarov, Head of Road Infrastructure Agency Mr. D. Pinjurov, Director Budget Directorate Ms. Eli Anavi, Ministry of Economy Mr. Plamen Oresharski, former Minister of Finance, current Member of Parliament Ms. Marinela Petrova, Director, Economic and Financial Policy Directorate, Ministry of Finance Mr. Roumen Porozhanov, State Fund Agriculture Mr. Dimitar Savov, Director of Policy, Ministry of Transport Mr. Angel Semerdjiev, SEWRC Chairman Ms. Valentina Simeonova, Deputy Minister of Labor and Social Policy Mr. Rosen Simitchiev, Consultant, BULGARTRANSGAZ, gas transmission infrastructure. Mr. Hristo Slavchev, former Deputy Head of Road Infrastructure Agency Ms. Angela Toneva, Member of SEWRC Mr. Krassimir Valchev, former Minister of Finance, currently Chief Secretary Ministry of Education Ms. Iglika Vassileva, former Ministry of Finance staff Ms. Petya Vassileva, Director, Coordination of EU Affairs Directorate, Council of Ministers Ms. Anna Yaneva, former Deputy Minister of Economy Ms. Boyan Zahariev, Open Society Institute CHILE Ms. Paula Darville, Director of Budgets of the Ministry of Finance (DIPRES) Mr. Luis Díaz, Ministerio de Planificación (MIDEPLAN) Mr. David Duarte, Dirección de Presupuestos (DIPRES) Ms. Pamela Gana, Superintendencia Pensiones Mr. José Pablo Gómez, DIPRES Mr. Andrés Hernando, MIDEPLAN Mr. Pedro Montt, Director, Planificación Min Desarrollo Social Mr. Felipe Saavedra, Director, Planificación Min Desarrollo Social Ms. Daniela Sugg, DIPRES Mr. Miguel Flores, Department Director, Subecretaría de Desarrollo Ms. Daniela Torre, Consejo Nacional de Educación 141 APPENDIX J PEOPLE INTERVIEWED Mr. Gerardo Uña, Ministerio Hacienda Ms. Marcia Varela, Consejo Nacional de Innovación para la Competitividad Mr. Herman von Gersdorf, DIPRES Mr. Ignacio Walker, Senator, Chilean Congress Ms. Isabel Zúñiga, DIPRES CHINA Mr. Liu Bin, Director, Ministry of Water Resources Ms. Liu, Policy Research Department, Ministry of Railways Ms. Liya Liu, Director, Institute of Comprehensive Transport Ms. Wang Miyao, Ministry of Health Mr. Lei Pingjing, Deputy Director General, Department of Tertiary Industry Ms. Shao, Foreign Capital Utilization Center, Ministry of Railways Mr. Bai Quan, Deputy Director, Energy Research Institute, National Development and Reform Commission Mr. Tang Chengcheng, Vice Director, Department of Traffic Engineering, Highway Research Institute Mr. Wang Xi, Ministry of Environmental Protection Mr. Wang Xin, Director, Foreign Exchange and Cooperation Department Mr. Zhang Zhengzhong, Hubei Development and Research Committee Carter Center for Project Supervision and Management Financial Stability Bureau Guangdong Finance Bureau Institute of Comprehensive Transport Ministry of Health Ministry of Human Resources and Social Security Ministry of Science and Technology State Information Center KAZAKHSTAN Ms. Ainur Abdyrakhmanova, Expert, Ministry of Labor and Social Protection Ms. Madina Abylkassymova, Vice Minister of Economic Development and Trade, Alternate World Bank Governor Mr. Kuat Akizhanov, State Borrowing Department Director, Ministry of Finance Mr. Baurzhan Bekeshev, Department Director, Ministry of Economic Development and Trade Mr. Erzhan Birzhanov, Head, Tax Service Modernization Division Ms. Zhannat Ertlessova, Advisor to Prime Minister Mr. Vyacheslav Evstafiev, Department Director Ms. Saule Gazizova, Department Director, Agency for Statistics Mr. Serik Irsaliev, Chairman, Committee for Education Quality, Ministry of Education and Science Mt. Bakhtiyar Issayev, Director, Oil and Gas Machine Building Department, KazMunayGas Ms. Ayajan Jangabylova, Division Head, Ministry of Labor and Social Protection Mr. Vagiz Khizmatullin, Director, Budget Policy Department, Ministry of Economic Development and Trade Mr. Farkhad Kuanganov, Deputy Governor for North Kazakhstan Oblast Mr. Zhardem Kurmangaziev, Head of Corporate Secretary Unit, Samruk-Kazyna Ms. Janel Kushukova, International Trade Department Director, Ministry of Economic Development and Trade Mr. Marat Kussainov, Vice Minister of Economic Development and Trade 142 APPENDIX J PEOPLE INTERVIEWED Mr. Argyn Kypshakov, Deputy Chairperson, Tax Committee Ms. Asel Makhanova, Acting Director, Department for Social Protection and Pensions Mr. Rakhim Oshakbayev, Deputy Chairman of the Board of Directors, Atameken Chamber of Commerce Ms. Irina Rubashina, Department Director, National Analytical Center Mr. Marat Sultangaziev, Director, Oil Revenue Management Department, Ministry of Finance Mr. Ruslan Sultanov, Director, Center for Trade Policy Development, Ministry of Economic Development and Trade Mr. Baktybek Tashenev, Director, Department for Methodology of State Asset Management, Ministry of Finance Mr. Bolat Tashmetov, Director, Procurement and Local Content Development Department Mr. Maksat Turlubayev, Division Head, Ministry of Economic Ms. Saule Umurzakova, Director, Department for Macroeconomic Analysis, Ministry of Economic Development and Trade Ms. Sholpan Zhakitova, Social Safety Department Director KUWAIT Mr. Adel Al-Bader, Assistant Undersecretary for Storage and Purchasing Systems, Ministry of Finance Capt. Al Haider, Deputy Director, Environment Public Authority Dr. Mahdy Al Jazaf, Vice President, Capital Market Commission Mr. Yousef Al Obaid, Executive Director of the Supervision Sector, Central Bank of Kuwait Dr. Khaled Al-Reshaid, Assistant Undersecretary for Planning and Information, Ministry of Education Ms. Aisha Al-Roudan, Executive Assistant Undersecretary for Planning and Information, Ministry of Education Mr. Dawood Al-Sabej, Ministry of Commerce and Industry Mr. Adel Al-Shallal, Government Performance Monitoring Agency, Al-Seif Palace Mr. Barrak Al-Sheetan, Assistant Undersecretary for General Accounting Affairs, Ministry of Finance Dr. Ades Al-Wugayan, Secretary General, Supreme Council for Planning and Ministry of Planning Mr. Abdulaziz Al-Zabin, President, Civil Service Commission Mr. Robert Beshcel, Prime Minister’s Office Mr. Ibrahim Daboub, Group Chief Executive Officer, National Bank of Kuwait H. E. Mr. Khalifa Hamada, Undersecretary, Ministry of Finance THE RUSSIAN FEDERATION Mr. Alexey Bakirey, Chairman of the Committee on Transport and Transit Policy, St. Petersburg Mr. Igor Bashmakov, Executive Director, Center for Energy Efficiency (CENEF), Moscow Mr. Sergey Belyaev, Director of Techology and Science, Russian Federation Research Institute for Water Management, Federal Agency for Water Resources Ms. Frolova Elena Borisovna, Head of Department of Cost-of Living Statistics Mr. Sergey Bubenov, Director of Economy and Finance, Ministry of Natural Resources Mr. Laikam Konstantin Emilievich, Deputy Director, Federal State Statistics Service (FSSS) Mr. Vsevolod Gabrilov, former Deputy Head of Department at Ministry of Natural Resources, current Head of Energy Efficiency and Natural Resources at SberBank, Moscow Ms. Ekaterina Kuvshinova, Department of Public-Private Partnerships, Committee for Investment and Strategic Projects, St. Petersburg Ms. Ksenia Kolesnikova, formerly Investment Director, Joint Stock Company, The Nadzemny Express (Nadex) Public-Private Patnership 143 APPENDIX J PEOPLE INTERVIEWED Ms. Natalia Larionova, Ministry of Economic Development Mr. Alexei Lavrov, Deputy Minister of Finance Mr. Vladislav Onishenko, Financial Recovery Plan Mr. Maxim Parshin, Deputy Director, State Governance Department, Ministry of Economic Development Ms. Nadezhda Prokhorova, Managing Director, Russian Federation Research Institute for Water Management, Federal Agency for Water Resources Mr. Alexander Puzanov, General Director, Institute of Urban Economics Mr. Gertsyi Selivanov, Federal Employment Service Mr. Andrei Sizov, Director, SovEcon Mr. Sergey Sukhov, Director of the Department for Regulation of Agricultural Market and Infrastructure Development, Ministry of Agriculture Mr. Mikhail Sukhov, Deputy Chairman of the Central Bank Mr. Sergei Sivaev, Director, Municipal Economy, Institute of Urban Economics SOUTH AFRICA Director of Planning Department, Gauteng Province Senior Advisor to the Ministry of Agriculture and Rural Development THAILAND Dr. Patchara Anuntasilpa, Ministry of Finance, Comptroller General's Department Dr. Chinnapat Bhumirat, Deputy Permanent Secretary, Office of Basic Education Commission Ms. Jirawan Boonperm, Permanent Secretary, Ministry of Information and Communication Technology Mr. Weerachai Chomsakorn, Director, Office of the Permanent Secretary, Prime Minister’s Office, Office of National Decentralization Committee Ms. Jutarat Chanvikul, Director, Bureau of Administrative System Development, Ministry of Labor Ms. Anoma Chattaris, Public Sector Development Officer, Office of the Public Sector Development Commission Mr. M.L. Patcharapakorn Devakula, Assistant Secretary-General, Office of Civil Service Commission Ms. Monvadee Juntima, Public Sector Development Officer, Office of the Public Sector Development Commission Dr. Kanitta, Office of Civil Service Commission Ms. Suwannee Khamman, Deputy Secretary-General, National Economic and Social Development Board Dr. Krissanapong Kirtikara, Advisor, King Mongkut's University of Thonburi (Retired) Dr. Sirilaksana Kohman, National Anti-Corruption Commission Dr. Warothai Kosolpisit, Ministry of Finance, the Fiscal Policy Office Ms. Lawan, Director of Tax Policy Bureau, Ministry of Finance, Fiscal Policy Office Mr. Kulasake Limpiyakorn, Division Director, Department of Comptroller General Dr. Supon Limwattananon, Senior Researcher, International Health Policy Program Dr. Rangsan Maneelek, Director, Bureau of Education Policy and Planning Ms. Mantanee, Office of the Prime Minister, Bureau of the Budget Ms. Saowakon Meesang, Revenue Department, Tax Policy and Planning Bureau Dr. Benjalug Namfa, Inspector General, Ministry of Education, Office of the Basic Education Commission (OBEC) Ms. Ratana Petchvijitra, Director of Bureau, Revenue Department Dr. Phusit Prakonsai, Director, International Health Policy Program Dr. Maka Phoojinda, Office of Civil Service Commission Dr. Pipawin, Office of Civil Service Commission 144 APPENDIX J PEOPLE INTERVIEWED Dr. Ratanawalee Rochanaroon, Ministry of Finance, Comptroller General's Department Dr. Yarika Ruengsiri, Revenue Department, Tax Policy and Planning Bureau Mr. Thanachoke Rungthipanonm, Ministry of Finance, Comptroller General's Department Dr. Chotima Sanguanphant, Public Sector Development Officer, Office of the Public Sector Development Commission Ms. Sumalee Satitchaicharoen, Director, Ministry of Finance, Fiscal Policy Office Dr. Petchsri Sirinirund, Director of National AIDS Management Office, Ministry of Public Health Mrs. Rarintip Sirorat, Deputy Permanent Secretary, Ministry of Social Development and Human Security Dr. Dechapiwat Na Songkhla, Office of the Prime Minister, Bureau of the Budget Ms. Suwatana Sripirom, Director, Fiscal Policy Office (currently at Mahidol University) Mrs. Chanohsuda Subunnarat, Public Sector Development Officer, Office of the Public Sector Development Commission Mr. Chantavit Sujatanound, Deputy Secretary General, Commission of Higher Education Mrs. Vunnaporn Devahastin Suthapreda, Executive Director, Office of the Public Sector Development Commission Dr. Viroj Tangcharoensathien, Director, International Health Policy Program Mr. Arkhom Termpittayapaisith, Secretary General of the Office of the National Economic and Social Development Board Ms. Chayanit Treetipvarakul, Revenue Department Ms. Khunying Kasama Varvarn, Former Permanent Secretary, Ministry of Education (retired) Dr. Suwit Wibulpolprasert, Senior Advisor on Disease Control, Ministry of Public Health Ms. Wilailuk Wisasa, Manager, International Affairs Cluster, National Health Security Office, Ministry of Social Development and Human Security Development Agencies CHILE Mr. Pablo González, Programa de las Naciones Unidas para el Desarrollo BULGARIA H.E.Mr.Karel van Kesteren, Dutch Ambassador H.E.Mr. J. Warlick, U.S. Ambassador Ms. Zinaida Zlatanova, Head of European Commission Representation in Bulgaria KAZAKSTHAN Ms. Sholpan Makhmudova, Program Specialist, U.S. Agency for International Development THE RUSSIAN FEDERATION Mrs. Guljahan Kurbanova, Economist, Food and Agriculture Organization, Regional Office for Europe and Central Asia 145 APPENDIX J PEOPLE INTERVIEWED Other BULGARIA Prof. Bistra Boeva, Ms. Snejana Dimitrova, La Fit Consultancy Prof. K. Kostadinov, Innovation expert Mr. Duhomir Minev, Institute of Sociology, Bulgarian Academy of Sciences Ms. Jeni Nacheva, former Director of Finance, National Health Insurance Fund Ms. Marinella Nenova, Bulgarian National Bank Mr. Julian Popov, Bulgarian School of Politics Mr. Tzvetan Simeonov, Chair, Bulgarian Chamber of Commerce and Industry Mr. Borislav Stefanov, Bulgaria Investment Agency Mr. Ruslan Stefanov and team, Center for the Study of Democracy Ms. Diana Stefanova, Chief Executive Officer, VMWare CHILE Mr. José M. Benavente, Universidad de Chile Ms. Heidi Berner, Universidad de Chile Mr. José J. Brunner, Centro de Políticas Comparadas de Educación Ms. Alejandra Contreras, Comisión Ingresa Mr. Cristián Cox, Universidad Católica Ms. Tania Hernández, Universidad de Chile Mr. Ignacio Irarrázabal, Centro de Políticas Públicas Mr. Patricio Meller, Corporación de Estudios para Latinoamérica Mr. Alejandra Mizala, Universidad de Chile Mr. Dagmar Raszinski, Asesorìas para el Desarrollo Ms. Andrea Repetto, Universidad Adolfo Ibañez Mr. Claudio Reyes, Facultad Latinoamericana de Ciencias Sociales Mr. Jorge Rodríguez, Corporación de Estudios para Latinoamérica Mr. Eduardo Saavedra, Universidad Alberto Hurtado Ms. Claudia Serrano, Centro Latinoamericano para el Desarrollo Rural Mr. Herman von Gersdorf, Dirección de Presupuesto (DIPRES) Mr. Juan Edo Vargas, Duhart, Larraín Vial Mr. Martin Zilic, Universidad Concepción CHINA Mr. Qi Bin, China Securities Regulatory Commission Ms. Lucy Chen, Global Health Center in Beijing University Mr. Huang Chengwei, Director General, International Poverty Reduction Center Mr. Zeng Gang, Shangai Stock Prof. Gao, Huajun, Beijing Normal University Mr. Zhang Huidong, Deputy Director, Training Department, International Poverty Reduction Center Madam Liang Jing, Chinese Institute of Certified Public Accountants Mr. Sun Lei, Director Financial Cooperatives Supervision Department—access to finance Mr. Bai Quan, Energy Efficiency Center, Energy Research Institute of National Development and Reform Commission Mr. Joseph Regan, Managing Director in Charge of China, J.P. Morgan Mr. Wang Xiaolin, Director, Research Department, International Poverty Reduction Center 146 APPENDIX J PEOPLE INTERVIEWED Mr. Shi Xiaoming, Director of Office for Non-Communicable Disease Prevention and Control, China Center for Disease Control Mr. Charlie Yie, Head of Global Markets, Standard Chartered Bank KAZAKHSTAN Ms. Almira Abenova, Chief Financial Officer, Deputy Chief Executive Officer, KazMicroFinance LLC Dr. Kanat Baigarin, Managing Director for Development, Nazarbayev University Mr. Pavel Beklemishev, General Director, Byelkamit JSC Mr. Aben Bektasov, Deputy Chairman, National Bank of Kazakhstan Mr. Aron Bernstein, Director, Bota Foundation (IREX) Mr. Kuandyk Bishimbayev, Deputy Chairman, Samruk-Kazyna Ms. Kadisha Dairova, Deputy Chairperson of the Executive Council, Nazarbayev University Mr. B.K. Damitov, Director for Institute for Education Quality Analysis, Kazakh National Pedagogical University Ms. K. Dairova, Vice Chairperson of the Executive Council, Nazarbayev University Ms. Zhanar Jandossova, President, Sange Research Center Ms. Aliya Janideyeva, Director Department for International Cooperation, Nazarbayev University Mr. Maksat Mamashev, Deputy Chairman for the Board, Nazarbayev University Ms. Ayana Manasova, Chairperson, Kasipkor Holding Mr. Nurlan Rakhmetov, Managing Director, Samruk-Kazyna Mr. Alexander Tikhonov, Lead Research Scientist, Nazarbayev University Ms. Nurilya Ussenova, Chairperson, Insolvency Committee, Ministry of Finance Ms. Khadizhat Uvaisova, Director,f Institute of Professional Appraisers of Kazakhstan National Bank: Committee for Financial Supervision, Department for Control and Supervision Methodology and Department for Financial Sustainability KUWAIT Dr. Ali Arifa, Kuwait University Dr. Jafar Al-Aryan, Kuwait University, Faculty of Social Sciences Dr. Salah Alghazali, Chairman, Kuwait Transparency Society Mr. Hussein bin Ali, Health, Safety, and Environment Specialist, Kuwait Gulf Oil Company (KGOC) Dr. Abdulghani Al Bazzaz, former Head of the Teacher Professional Development Unit Dr. Abla Al-Eissa,Training and Development Department, Jabriya Dr. Yousef Al-Ebraheem, Advisor, Diwan Amiri Mr. Hamed Ali Al-Hasawi, Secretary General, Kuwait Banking Association Dr. Ridha Al-Khayyat, Director, National Center for Education and Development Babtain Dr. Ali Al-Mudaf, Director General, Public Authority for Industry Mr. Wael Al-Qalaf, Data Analysis Specialist, Jabriya Dr. Quasem Al-Sarraf, Kuwait University, Faculty of Education and Psychology Dr. Fawzi Al-Sultan, consultant THE RUSSIAN FEDERATION Mr Anatoliy Aksakov, President, Association of Regional Banks Mr. Bashmakov, Executive Director, Center of Energy Efficiency Mr. Rudolph Bulavin, Advisor, Robert Gordon University Mr. Tatyana Chugunova, Central Bank Prof. Victor Danilov-Danilyan, Institure of Water Problems Mr. Vsevolod Gavrilov, Sberbank Ms. Galina Kurlyandskaya, Center for Fiscal Policy 147 APPENDIX J PEOPLE INTERVIEWED Ms. Antonina Kovalevskaya, Center for Fiscal Policy Mr. Alexander V. Korbut, Vice President, Robert Gordon University Mr. Leonid Levkovich -Maslyuk, Russian Venture Company Ms. Lilia Ovcharova, Director, Independent Institute for Social Politics Mr. Alexander Puzanov, Executive Director, Institute for Urban Economics Mr. Alexander Pirozhenko, Agency of Strategic Initiatives Mr. Semenyaka, Head, Mortgage Agency Mr. Andrei A. Sisov, Managing Director, SovEcon Mr. Richard Warburton, Chief Executive Officer, Black Earth Farming (Agro-Invest), Ltd. (Swedish- Russian Joint venture) Mr. Andrey Yakovlev, University Higher School of Economics Mr. Arkady Zlochevsky, President, Russian Grain Union (Robert Gordon University premises) Ms. Natalia Zubarevich, Head of Regional Programs, Ind. Institute for Social Politics Representatives from United Grain Corp. and Summa Group (holding company) Federal Anti-Monopoly Service, Solontsova L.V. Sadovaya Kudrinskaya Federal Financial Market Service SOUTH AFRICA Ms. Johann Kirsten, Chair, Department of Agricultural Economics, University of Pretoria Ms. Ruth Hall, Director, Institute for Poverty, Land, and Agrarian Studies, University of the Western Cape Mr. David Mayson, Principal, Phuhlisani (Mr. Rick de Satge, Specialist, Phuhlisani THAILAND Ms. Karyn Kaplan, Director for Policy and Development, Thai AIDS Treatment Action Group Mr. Kulasake Limpiyakorn, Ministry of Finance, Department of Comptroller General Dr. Supon Limwattananon, International Health Policy Program, Ministry of Public Health Dr. Rangsan Maneelek, Ministry of Education Dr. Benjalug Namfa, Ministry of Education Mr. Phusit Prakonsai, Ministry of Public Health Mr. Phusit Prakonsai, Ministry of Public Health Dr. Thaworn Sakunphanit, Deputy Director, Infectious Disease Institute Dr. Ammar Siam-Walla, Distinguished Scholar, Thailand Development Research Institute (TDRI) Dr. Jaruayporn Srisasalux, Research Manager, Health System Research Institute Dr. Samrit Srithamrongsawat, Director, Health Insurance System Research Office, Infectious Disease Institute Dr. Chantavit Sujatnound, Commission Higher Education Dr. Viroj Tangcharoensathien, International Health Policy Program, Ministry of Public Health Mr. Pascal Tanguay, Program Director, Population Services International Dr. Sakon Varanyuwatan, Thammasat University Dr. Maitri Wasuntiwongse, National Consultant Mr. Wilailuk Wisasa, National Health Security Office 148 References China Securities Regulatory Commission. 2008. China Capital Markets Development Report. Almanac of China's Finance and Banking Editorial Board. Beijing: China Securities Regulatory Commission. Clarke, George R.G., James Habyarimana, Michael Ingram, David Kaplan, and Vijaya Ramachandran. 2007. South Africa: An Assessment of the Investment Climate. Washington, DC: World Bank. Hausmann, Ricardo, Akash Deep; Rafael Di Tella; Jeffrey Frankel, Robert Lawrence; Dani Rodrik; and Andrés Velasco 2011. “Growth and Competitiveness in Kazakhstan: Issues and Priorities in the Areas of Macroeconomic, Industrial, Trade and Institutional Development Policies.” Working Papers, Center for International Development, Harvard University, Cambridge, Mass. Hawkins, Loraine, Jaruayporn Srisasalux, and Sutayut Osornprasop. 2009. Devolution of Health Centers and Hospital Autonomy in Thailand: A Rapid Assessment. Washington, DC: World Bank. Hinz, Richard P., Asta Zviniene, and Anna-Marie Vilamovska. 2005. The New Pensions in Kazakhstan: Challenges in Making the Transition. Social Protection Discussion Paper No. 0537. Washington, DC: World Bank. IEG (Independent Evaluation Group). 2007. Development Results in Middle-Income Countries: An Evaluation of the World Bank’s Support. Washington, DC: World Bank. ———. 2008. Using Knowledge to Improve Development Effectiveness: An Evaluation of World Bank Economic and Sector Work and Technical Assistance, 2000–2006. Washington, DC: World Bank. ———. 2009. Independent Evaluation of IFC’s Development Results: Knowledge for Private Sector Development. Washington, DC: World Bank. ———. 2010. World Bank economic reports on growth diagnostics in four African countries : Ghana, Mauritius, Nigeria, and Uganda. Performance Assessment Review. Washington, DC: World Bank. ———. 2011. Performance Assessment Review of Investment Climate Assessments in Five Transforming Economies: Bangladesh, Egypt, Guatemala, Kenya, and Vietnam. Washington, DC: World Bank. ———. 2011. World Bank Support for Revenue Policy Reform in Eastern Europe and Central Asia: With Performance Assessment Reports of ESW in Georgia, Kazakhstan and Kyrgyz Republic. Washington, DC: World Bank. ———. 2012. Results and Performance of the World Bank Group 2012. Washington, DC: World Bank. IFC (International Finance Corporation). 2012. IFC Secured Transactions Advisory Project in China. Washington, DC: World Bank IMF (International Monetary Fund). 2008. “Thailand: Financial System Stability Assessment.” IMF Country Report No. 09/147. Washington, DC: IMF. ———. 2010. “Russian Federation: Financial Sector Stability Assessment Update.” IMF Country Report No. 10/96. Washington, DC: IMF. ———. 2010. “Kuwait: Financial System Stability Assessment—Update.” IMF Country Report No. 10/239. Washington, DC: IMF. ———. 2011. “Chile: Financial System Stability Assessment.” IMF Country Report No. 11/261. Washington, DC: IMF. 149 REFERENCES ———. 2011. “China: Financial System Stability Assessment.” IMF Country Report No. 11/321. Washington, DC: IMF. Organisation for Economic Co-operation and Development (OECD). 2007. Reviews of National Policies for Education: Higher Education in Kazakhsta. OECD: Paris. Revenga, Ana, Mead Over, Emiko Masaki, Wiwat Peerapatanapokin, Julian Gold, Viroj Tangcharoensathien, and Sombat Thanprasertsuk. 2006. The Economics of Effective AIDS Treatment: Evaluating Policy Options for Thailand. Washington, DC: World Bank. World Bank. 2003. Thailand—Country Assistance Strategy for FY03-FY05. Washington, DC: World Bank. ———. 2005. “The Road to Successful EU Integration.The Policy Agenda Poverty Reduction and Economic Management Unit. Europe and Central Asia Region. Country Economic Memorandum.” Washington, DC: World Bank. ———. 2006. “Thailand Investment Climate, Firm Competitiveness and Growth.” Washington, DC: World Bank. ———. 2006. Country Partnership Strategy for the Russian Federation for Period FY2007-FY2009. Report No. 37901-RU. Washington, DC: World Bank. ———. 2006. Russian Federation—Institutional Reform in Russia: Moving from Design to Implementation in a Multi-Level Governance Context. Washington, DC: World Bank. ———. 2007. Bulgaria—Accelerating Bulgaria's Convergence: The Challenge of Raising Productivity. 2 vols. Washington, DC: World Bank. ———.2007. Chile—Country Partnership Strategy for the Period 2007-2010. Washington, DC: World Bank. ———. 2007. Republic of Kazakhstan—Report on the Observance of Standards and Codes (ROSC): Accounting and Auditing. Washington, DC: World Bank. ———.2007. “South Africa Enhancing the Effectiveness of Government in Promoting Micro, Small and Medium Enterprise.” Washington, DC: World Bank. ———.2007. South Africa—Country Partnership Strategy for the Period 2008-2012. Washington, DC: World Bank. ———.2007. An Assessment of the Investment Climate in South Africa. Washington, DC: World Bank. ———. 2008. “Chile: Estudio de Evaluación en Profundidad del Programa de Mejoramiento de la Gestión (PMG).” Vol. 1. Washington, DC: World Bank. ———. 2008. Bulgaria -- Resourcing the Judiciary for Performance and Accountability: A Judicial Public Expenditure and Institutional Review. Washington, DC: World Bank. ———. 2008. Bulgaria—Country Partnership Strategy for the Period 2007-2009. Washington, DC: World Bank. ———. 2008. Thailand—Investment Climate Assessment Update. Washington, DC: World Bank. ———. 2009. “Financial Sector Assessment Republic of Kazakhstan.” Washington, DC: World Bank. ———. 2009. “Institutional Framework for Early Childhood Development in Chile.” Policy Note. Report No. 71693. Washington, DC: World Bank. ———. 2009. Bulgaria Social Assistance Programs: Cost, Coverage, Targeting and Poverty Impact. Washington, DC: World Bank. ———. 2009. Bulgaria: Living Conditions before and after EU Accession. Human Development Sector Unit and Ministry of Labor and Social Policy. Washington, DC: World Bank. 150 REFERENCES ———. 2009. Bulgaria—Railways Policy Note. Washington, DC: World Bank. ———. 2009. Case Study on the Management of Chile’s Social Security Reform: A Client-Focused, Public- Private System with the Support of IT Solutions. Washington, DC: World Bank. ———.2009. China—Report on the Observance of Standards and Codes (ROSC)—Accounting and Auditing. Washington, DC: World Bank. ———. 2009. Malaysia Productivity and Investment Climate Assessment Update. Washington, DC: World Bank. ———. 2009. Russian Federation Regional Development and Growth Agglomerations—The Longer Term Challenges of Economic Transition in the Russian Federation—A Country Economic Memorandum for Russia. Washington, DC: World Bank. ———. 2010. “Kuwait Public Expenditure and Financial Accountability Public Financial Management Performance Assessment Report.” Washington, DC: World Bank. ———. 2011. “The State of World Bank Knowledge Services: Knowledge for Development 2011.” Washington, DC: World Bank. ———. 2011. “La agenda de la descentralización en Chile: Propuestas de corto y mediano plazo para mejorar la gestión, fortalecer la rendición de cuentas y asegurar el control de resultados en el nivel municipal.” Washington, DC: World Bank. ———. 2011. Bulgaria—Country Partnership Strategy for 2011-2013. Washington, DC: World Bank. ———. 2011. Chile—Country Partnership Strategy for the Period FY11-FY16. Washington, DC: World Bank. ———. 2011. Chile's State-Guaranteed Student Loan Program: Analysis and Evaluation. (CAE). Washington, DC: World Bank. ———. 2011. Improving the Business Environment for Growth and Job Creation in South Africa: The Second Investment Climate Assessment. Washington, DC: World Bank. ———.2011. Malaysia: Moving up the Value Chain—A Study of Malaysia’s Solar and Medical Device Industries. Washington, DC: World Bank. ———.2011. Public Sector Reforms and Human Resources for Health in Thailand: An Exploration of Impacts, Issues and Options for Moving Forward. Washington, DC: World Bank. ———.2011. Reducing Inequality for Shared Growth in China: Strategy and Policy Options for Guangdong Province. Washington, DC: World Bank. ———.2011. Russian Federation—Country Partnership Strategy for the Period 2012-2016. Washington, DC: World Bank. ———.2011. Improving the Business Environment for Growth and Job Creation in South Africa: the Second Investment Climate Assessment. Washington, DC: World Bank. ———.2011. Harm Reduction Policies and Interventions for Injection Drug Users in Thailand. Washington, DC: World Bank. ———.2012. Bulgaria—Public Expenditures for Growth and Competitiveness. Washington, DC. ———.2012. China 2030: Building a Modern, Harmonious, and Creative High-Income Society. Washington, DC: World Bank. ———.2012. Integrated Air Pollution Management in China: Developing Particulate Matter Control. Washington, DC: World Bank. 151 REFERENCES ———.2012. Kazakhstan—Country Partnership Strategy for the Period FY12-FY17. Washington, DC: World Bank. ———.2012. Leading with Ideas: Skills for Growth and Equity in Thailand. Washington, DC: World Bank. ———.2012. Malaysia: Report on the Observance of Standards and Codes (ROSC), Corporate Governance Country Assessment. Washington, DC: World Bank. ———.2012. The Basel III Financial Architecture and Emerging Regulatory Developments in Macro Prudential Tools. Washington, DC: World Bank. ———.2012. The Provision of Reimbursable Advisory Services. Operational Memorandum. Washington, DC: World Bank. World Bank and Department of Land Affairs (South Africa). 2007. “Accelerating Sustainable, Efficient and Equitable Land Reform: Case Study of the Qedusizi/Besters Cluster Project.” Africa Region Working Paper Series No. 109. Washington, DC: World Bank. World bank and Foreign Investment Advisory Service (South Africa). 2006. Sector Study of the Effective Tax Burden: South Africa. Washington, DC: World Bank. World Bank and International Finance Corporation. 2012. “Kazakhstan Enterprise Survey.” Washington, DC: World Bank. World Bank and Ministry of Public Health (Thailand). 2010. Revitalising HIV prevention in Thailand: a critical assessment. Final report. Washington, DC: World Bank. World Bank and Office of the National Economic and Social Development Board (Thailand). 2008. “Towards a Knowledge Economy in Thailand.” Washington, DC: World Bank. Yusuf, Shahid, and Kaoru Nabeshima. 2009. Tiger Economies Under Threat: A Comparative Analysis of Malaysia’s Industrial Prospects and Policy Options. Washington, DC: World Bank. Yusuf, Shahid. 2010. People's Republic of China: China PSD Program. Washington, DC: World Bank. Zhang, Chunlin, Douglas Zhihua Zeng, William Peter Mako, and James Seward. 2008. Promoting Enterprise-Led Innovation in China. Washington, DC: World Bank. 152 Endnotes 1The Bank has continued to improve its internal organization for knowledge management, and is presently engaged in a reinvigoration of the peer review process to improve the quality of knowledge products. 2“Core knowledge services” include economic and sector work (ESW) and technical assistance (TA) for external clients, knowledge services produced as a public good, and analytical work for internal use. 3 International Finance Corporation (IFC) Annual Reports, FY06, FY08, and FY12. 4 Over FY06-11 Reimbursable Advisory Services grew from $12.2 million to $30.6 million. 5Such “knowledge flows from operations” were estimated by the Independent Evaluation of IFC’s Development Results: Knowledge for Private Sector Development study at $2.5 billion in FY11, far outstripping the estimated amount of core Bank knowledge services ($606 million, of which $444 million was for external clients). 6 Infrastructure includes energy and mining, environment, transport, and urban development. Poverty reduction/social protection also includes social development. Private sector includes private and financial sector and financial management. 7 The Independent Evaluation Group (IEG) conducted an evaluation of World Bank ESW and TA in 2008 (Using Knowledge to Improve Development Effectiveness: An Evaluation of World Bank Economic and Sector Work and Technical Assistance, 2000–2006). An evaluation of growth diagnostics in Africa was conducted in 2010 (Performance Assessment Review, World Bank Economic Reports on Growth Diagnostics in Four African Countries: Ghana, Mauritius, Nigeria, and Uganda), followed by an evaluation of AAA on Revenue Policy (World Bank Support for Revenue Policy Reform in Eastern Europe and Central Asia: Performance Assessment Reports of ESW in Georgia, Kazakhstan and Kyrgyz Republic) and a clustered evaluation of investment climate assessments in emerging economies (Performance Assessment Review of Investment Climate Assessments in Five Transforming Economies: Bangladesh, Egypt, Guatemala, Kenya, and Vietnam). 8 In 2010 the Bank elaborated an Interim Strategy Note for FY11-12, as a result of the 2008-09 global financial crisis that led to a resumption of lending to Thailand with a Development Policy Loan of $1 billion. 9 The criteria are as follows: client need and ownership or market failure; intended development results; the distinct value IFC brings considering roles and activities by other stakeholders including other Bank Group institutions; strategic relevance, which includes fit with IFC and Bank Group strategies; links to other IFC projects; and risks. 10 The Joint Economic Research Program (JERP) is a demand-driven, cofinanced program, whereby the government and the Bank annually agree on AAA to be conducted. The JERP grown from $1.3 million in 2004 (of which 40 percent was government financed) to $4.3 million in 2012 (of which 85 percent was government financed). 11 The project was requested by the Competition Commission and Tribunal of South Africa, but the end-product was not to the liking of counterparts perhaps reflecting weak communications during project preparation as well as changes in counterparts who came with new priorities. 12 Many IFC projects do not target policy makers, including, for example, programs supporting private sector entrepreneurship in South Africa, which mainly target private providers of programs and entrepreneurs. 13IEG is aware that in August 2012 the Bank appointed a liaison officer based in Chile to strengthen the dialogue on medium-term development issues in the country and monitor the program closely with the government. 14 The Bank’s three knowledge roles were analyzed in “The State of World Bank Knowledge Services: Knowledge for Development 2011.” 15 IEG understands that the Bank has never taken the policy position that it can or should act as a consultancy. In this report, the authors refer to the Bank sometimes becoming a de facto consultant to the government under a reimbursable knowledge service contract. For the definition of the Bank’s role in the provision of reimbursable 153 ENDNOTES services. IEG acknowledges the inherent tension in remaining engaged with the development agenda while addressing the short-term priorities of clients under RAS. There are practical limits in a difficult budget environment for a business model that would require resources and generate no income, and the Bank could evaluate with clients essential strategic areas to be covered outside the RAS program. 16 IFC additionality is the unique benefit or addition of value that IFC brings to a project, and that the client would not receive without IFC intervention. 17By way of examples, the report covered Chile’s policy to successfully prevent real appreciation of the exchange rate and therefore protect competitiveness, South Korea’s use conglomerates for development, Kyrgyzstan flat rate tariff policy, and China’s use of the World Trade Organization to enhance domestic reforms. 18 However, the IFC Access to Finance project was an important input to the issuance of a widely disseminated IFC paper “Promotion of SME: A Review of International Experience.” Similarly, the two South Africa projects on small and medium enterprises (SME) tax burden and compliance costs focused on generating country information through surveys and not only international comparisons, practices, and experience; but the Advisory Services were part of larger effort in the Africa region that would enable comparison and analysis of different country practices. 19Key recommendations were: introducing lump sum (“block”) grants to local governments with amounts that usually take into account spending needs, fiscal capacities, fiscal efforts of local governments, and other factors; basing rural project selection on a participatory planning and budgeting process, including participation in identifying and prioritizing local problems, finding solutions and preparing project proposals at the village level, and making final decision at the county level; developing monitoring and evaluation processes, instruments, systems, and funding; and reducing duplication among various funding programs by deriving projects from county-level comprehensive plans and allowing counties to use funds from other programs or sources for the counterpart funding requirement. 20The study submitted three options, with a detailed discussion of each: to strengthen the verification of household means, to replace the unverified income test with a proxy-means test, and to develop a hybrid means test. 21 The share of investment lending operations for which IEG rated the development outcome moderately satisfactory or higher was 78 in FY06-08 and 70 percent in FY09-11 (projects exiting the active portfolio). The share of Development Policy Operations with a development outcome rating of moderately satisfactory or higher was 80 in FY06-08 and 83 percent in FY09-11. See Results and Performance of the World Bank Group 2012. 22Although in some cases the client did not take the Bank’s advice, its knowledge was deepened. Indeed, in a number of cases clients requested knowledge services to inform an ongoing policy discussion. 23 IEG is aware that in many instances brainstorming exercises are catered to a broad and diverse group of stakeholders. At the same time, the IEG team had a difficult time finding public minutes or notes of the results of a number of brainstorming sessions. IEG also recognizes that the Bank team working on Kazakhstan is developing formal results frameworks as part of the programmatic JERP structure and new country partnership strategies results framework. 24 IEG is aware that since FY11 the Bank has adopted a three-pronged approach to improve results and measurement of knowledge services: (i) strengthening self-assessments; (ii) implementing client feedback instruments to elicit client assessment of the usefulness and relevance of our knowledge products; and (iii) ex- post review by external evaluators (IEG is analyzing how it could be part of this system). In this vein, the Operations Policy and Country Services department is encouraging operational staff to structure knowledge activities around clear monitoring indicators based on a theory of change. 25 IEG understands that in recent years IFC introduced reforms to improve Advisory Services implementation: peer review and quality at entry, client agreement for all Advisory Services projects, detailed guidelines for client contributions, standardized results measurement frameworks, bi-yearly project supervision, annual client surveys, and project completion reports. 26 See also IEG Review of CPSCR for China, 2012. 154 ENDNOTES 27 IEG understands that the Bank is in the process of improving the internal peer review process to ensure good technical quality of knowledge services. 28IEG understands that the Bank is applying the three-pronged monitoring approach (see endnote 24) to RAS, and developing more detailed guidance for staff application of operational policies to the RAS program. 29See Operational Memorandum on “The Provision of Fee-Based Services,” updated in 2008 and proposed update in October 2012. Indirect and overhead costs have been approximated in the past by a factor of 50 percent of direct costs, unless actual estimates are available. 30The Bank has ample presence in Kazakhstan: a regional office in Almaty and a country office in Astana, where the country manager resides. 31IEG recognizes that the Bank has contributed to improving the public financial management system, public administration, and delivery of key public services. Work is continuing in these areas under the new country partnership strategy. The Bank expects results on public administration reform, which have been slow to come so far. 32IEG understands that there is ongoing economic sector work under the FY13 JERP that looks specifically into the issue of countercyclical fiscal rules applied in other resource-rich countries. 155