The Effectiveness of Private Sector Development Interventions in Fragile and Conflict-Affected Situations: Evidence from Evaluations November 2016 Chaoying Liu & Emily Harwit Development Impact Department – IFC, World Bank Group List of Abbreviations A2F Access to Finance AREU Afghan Research and Evaluation Unit AUSAID Australian Government’s overseas aid program BE Business Edge+ DAC Development Assistance Criteria DCED Donor Committee for Enterprise Development DCOF Displaced Children and Orphans Fund – a USAID funded program DFID U.K. Department for International Development DNH Do no harm DRC Democratic Republic of Congo EU European Union FARC Revolutionary Armed Forces of Colombia FCS Fragile and Conflict Situations GIZ Deutsche Gesellschaft für Internationale Zusammenarbeit GON Government of Norway IC Investment Climate IDEV Independent Evaluation Department of the AFDB IDP Internally Displaced Person/ People IEG Independent Evaluations Group (of the World Bank) IFC International Finance Corporation INF Infrastructure projects IRC International Rescue Committee JICA Japan International Cooperation Agency M&E Monitoring & Evaluation NGO Non-governmental organization OECD Organization for Economic Co-operation and Development PNG Papua New Guinea PPP Public Private Partnership PSD Private Sector Development RAMSI Regional Assistance Mission to the Solomon Islands – funded by AUSAID RCT Random Control Trial SME Small and Medium-sized Enterprise TA Technical Assistance UNIDO United Nations Industrial Development Organization UNDP United Nations Development Programme USAID United States Agency for International Development VSLA Village Savings and Loan Association WBG World Bank Group WDR World Development Report (a publication of the World Bank) i Table of Contents List of Abbreviations .................................................................................................................... i Table of Contents ......................................................................................................................... ii Executive Summary ..................................................................................................................... 1 Introduction .................................................................................................................................. 4 Evidence of effectiveness of PSD interventions in FCS ..................................................... 4 Objectives ............................................................................................................................ 5 Methodology ....................................................................................................................... 6 Limitations .......................................................................................................................... 9 Key Findings .............................................................................................................................. 10 Effectiveness and value of PSD interventions .................................................................. 10 What creates success? ....................................................................................................... 12 What types of interventions were most likely to lead to results? ...................................... 13 Contribution of PSD interventions to peace and stability ................................................. 22 Quality of the evaluations ................................................................................................. 25 Key challenges and constraints in conducting evaluations ............................................... 26 Conclusions and Implications .................................................................................................... 27 Annexes...................................................................................................................................... 30 Annex 1. Country typology ............................................................................................... 31 Annex 2. Definitions—effectiveness and evaluability...................................................... 32 Annex 3. Review methodology ......................................................................................... 34 Annex 4. A list of evaluations reviewed ........................................................................... 35 ii Executive Summary This systematic review is an effort to fill the knowledge gap about the effectiveness of Private Sector Development (PSD) interventions in Fragile and Conflict-Affected Situations (FCS). The objective of the review is to identify and extract evidence from published evaluations of PSD interventions in FCS on what has or has not worked in terms of achieving development results, including contributions to peace and stability. The review identified 312 published evaluations of PSD interventions carried out between 2005 and 2014, of which 56 constituted the final data set for the review analysis. The review covered evaluations in 23 countries classified as FCS by the World Bank from 2005-14 and three other countries that experienced conflict. Annex 1 shows how each of the 23 countries were categorized according to the country’s conflict status from the FCS list. In summarizing the evidence, we defined ‘effectiveness’ as how external evaluators measured the degree of success in attaining the planned results and objectives of PSD projects in FCS. Project effectiveness was measured within four business lines: SME support, infrastructure, access to finance and investment climate reform. The evaluability, or the ability of evaluators to determine how well projects were implemented, was weak in some projects under review. For example, in 25 percent of the evaluated projects, outcomes were either poorly defined or not appropriate. In addition, the basis for determining success across individual projects was not always clear because projects sometimes defined outputs and outcomes differently, even when long and short-term results were achieved. This limited our ability to appropriately catalog the projects’ evidence in a consistent and clear manner. An explanation for how we defined and cataloged degrees of success related to project effectiveness in the evaluations is provided in Annex 2. Key Findings Some important findings were identified from the review of the 56 evaluations:  A majority of projects (84 percent) were rated effective in achieving expected deliverables/outputs, while fewer than 50 percent were successful in reaching and exceeding planned outcomes and impact, for example, in job creation or investment generation;  Projects supporting small and medium-sized enterprises (SMEs) were the most successful within the four business areas’ programs analyzed: evaluators rated 58 percent of these as effective. Interventions to create infrastructure closely followed, with 57 percent rated as being effective. Projects to improve business environment and support intermediaries in financial sectors had the lowest overall ratings for effectiveness, at 31 and 30 percent respectively; 1  Successful PSD projects supporting SMEs involved training of entrepreneurs for livelihood development, rehabilitation of micro or small businesses, and improving access to markets. Business support along the entire supply chain was key to success;  Projects involving direct investments had the highest rating for effectiveness (75 percent) as compared to those employing technical assistance alone (44 percent) or those mixed with grants (60 percent);  There was no clear evidence that the reviewed PSD interventions contributed directly to peace and stability. Nevertheless, the review noted that conflict and vulnerability sensitivity analyses which included a peace-building or “do no harm” approach, seemed to enhance the overall positive results and effectiveness of projects or programs. Critical factors for the success of PSD interventions in FCS were observed as follows:  Identifying sectors that show higher potential for greater development results and applying the appropriate instrument in that sector seems to be key for success;  Providing financial support through investments is critical for countries to cope with or recover from conflict;  Involving investment and investors helps ensure project success, particularly if donors or companies have invested their own funds into a project (particularly in banks or in infrastructure projects);  Engaging local communities in project design and implementation is important, including vulnerable and at-risk populations who can help develop creative solutions for local conditions;  Starting with small, pilot projects and building on those that work can lead to success on a larger scale, across the country or sector;  Analyzing and factoring in fragility and conflict as part of project design work is needed, particularly in trying to assist vulnerable populations affected by the conflict;  Engaging local ‘champions’ or steering committees throughout implementation who support project monitoring will help to ensure that projects are on track, and are steered toward success; and  Addressing sustainability from the outset and ensuring that plans are achievable and realistic within a short period of time is a key design element for all projects in FCS, particularly for projects of 2-3 years. While the less-than-50 percent success rate for achieving planned outcomes is not surprising, considering the extraordinary challenges of working in countries which are still at war, recovering from conflict, or where governments are unstable and fragile, the review noted some specific and recurring factors that contributed to failure: (i) poor project design; (ii) lack of investment; and (iii) insufficient engagement of the private sector, which was instrumental in achieving both the planned outputs and the project outcomes. 2 Implications The findings may have implications for both institutions and program leaders as well as for evaluation community and evaluation practitioners. Given the high priority of FCS in the development agenda, it is imperative for the evaluation community to establish more and credible evidence about the effectiveness of PSD interventions to improve our understanding of their value and potential contribution to peace and stability. Currently, few rigorous evaluations of PSD interventions in the FCS space are available. The evaluation of a USAID project in Burundi using the Random Control Trial (RCT) method provides a good example for robust evaluation measures, which demonstrates that rigorous evaluation in FCS is possible. While some FCS-related factors, such as security, may be inevitable when carrying out evaluations in FCS settings, many important factors can be addressed, including engaging the project team earlier and working with the team to ensure the program/project log frame is well established and indicators are clearly defined. It is also useful to involve local communities and/or beneficiaries in the design of monitoring and evaluation systems (M&E). When M&E is a central part of project planning, longer-term sustainability and effectiveness of an intervention improves. When people and communities are engaged in active monitoring of projects, action can be taken when results go off-track. Stakeholders will also be more engaged in ensuring the sustainability of a project. Finally, there is a great need for experimenting and adopting new and innovative approaches to evaluation, for example, by using technology to address challenges faced in the FCS setting. 3 Introduction The 2011 World Development Report (WDR) clearly defines problems facing FCS countries today. It presents a model of the cyclical nature of violence that has kept approximately 340 million people captive in repeated cycles of war. This has led to population displacement, lost incomes, and an inability to escape poverty. A key point in the report is that “21st century conflict and violence are a development problem that does not fit the 20th century mold.”1 Evidence of effectiveness of PSD interventions in FCS The donor and development community believes that the private sector has an important role to play in fragile situations, both conflict and post-conflict, by providing goods and services that an often dysfunctional public sector cannot. The underlying theory of this belief is supported by the work of many academic researchers and practitioners. There is a large body of publically-available literature on the role of such PSD interventions. For example, the Donor Committee for Enterprise Development (DCED) has established an online library which has collected more than 450 books, journal articles, and tools/guidelines about PSD in FCS. In a key reference published by DCED, a review of private sector development in FCS, the author examines various approaches that DCED’s member agencies and others have adopted towards post-conflict PSD. Based on empirical experience, the review concludes that “It is now being more widely recognized that private sector development has a crucial role to play in post-conflict, and other conflict-affected, situations. However, there is still some uncertainty as to what distinguishes post-conflict PSD from PSD in other situations and how post-conflict PSD should be carried out.” The author also points out that “The value of these different approaches would be better understood if the results of post-conflict PSD programming could be assessed more accurately. To date, there has been little work done on assessing the impact of programming in post-conflict situations.”2 Another systematic review of literature on employment in fragile states indicates that “many studies and policy documents being based on the [perfectly logical] assumption that employment creation will promote poverty reduction and stability.… there are numerous empirical studies which decompose the key determinants of poverty reduction in non-fragile states, and employment is consistently identified as a major factor in a range of different country contexts, …but similar studies were not identified in fragile states.” It concludes that “despite the centrality of employment creation as an instrument to promote stability in the fragile states policy discourse, no robust qualitative or quantitative evidence was found to illustrate this relationship in the literature.”3 1“The World Development Report 2011: Conflict, Security and Development”; The World Bank; Washington DC, 2011. 2“Private Sector Development in Post-Conflict Countries: A Review of Current Literature and Practice”; Naoise Mac Sweeney; Cambridge UK; 2008. 3 “Fragile States 2014: Domestic Revenue Mobilisation in Fragile States;” OECD; Holmes et al, 2014. 4 In summary, despite an increase in investments from international communities to support PSD efforts in FCS, and many cases of successful interventions, there has been no consensus over what constitutes “best practice” in PSD interventions and there is no solid evidence on which types of PSD interventions (instrument and modality) did or did not work in FCS environments. Specifically, several gaps that need to be filled by further research and studies, include the establishment of solid evidence of effectiveness of PSD interventions in terms of:  Understanding of effectiveness of various PSD instruments/tools and modalities going beyond technical assistance programs or capacity building;  Understanding conditions and factors that enable effective PSD interventions; and  Understanding challenges in conducting rigorous evaluations in FCS environments. Older evaluations tend not to have such comprehensive assessment criteria. They do not differentiate between different types of instruments and rarely discuss their data source and data quality issues explicitly. Recent studies and evaluations are more likely to be more rigorous in that respect. Therefore, this study benefits from a body of literature that was developed after the 2008 systematic review of PSD in FCS. Objectives This review was carried out in an effort to fill the knowledge gap about the effectiveness of PSD interventions in FCS. The objective of the analysis is to identify and extract evidence from evaluations from a range of donors and institutions related to PSD interventions in FCS to observe broader themes on what has or has not worked in terms of achieving development results, including contributions to peace and stability. To the extent possible, the review aims to identify and summarize evidence and lessons that will help:  Institutions make smarter decisions in selecting the appropriate instruments to be used in FCS;  Project teams design and implement successful PSD interventions; and  Evaluators develop and conduct robust evaluations in FCS environments. In addition, the review is expected to contribute to a growing knowledge base to be used by researchers and practitioners from other international development agencies involved in designing and implementing PSD interventions in FCS. 5 Methodology Key review questions Considering key areas and instruments in International Finance Corporation (IFC) business operations, the review sought answers to key questions pertaining to three important themes that aim to fill the knowledge gap identified in the previous section. These themes and the relevant review questions are: (i) Value and effectiveness of PSD interventions  Is there evidence that the identified interventions achieve their intended program objectives? The review will pay particular attention to key impact measures of job creation and investment generation.  Where projects were effective, what contributed to their success? What factors (such as types of FCS countries) and characteristics (such as types of instruments, and sequencing) influenced the achievements of these PSD interventions? (ii) Contribution of PSD interventions to peace and stability  Is there evidence that the identified interventions contributed to peace and stability? If yes, then how?  To what extent were “do-no-harm” principles taken into account in design and implementation of identified PSD interventions? (iii) Quality of the evaluations  What is the quality of the evaluations?  What were the key challenges and constraints affecting the quality of the evaluation? 6 Figure 1: Conceptual framework of the review Peace and Stability Contribution Economic Development Influence factors Contribution • Type of FCS countries • Type of instruments Improving Supporting Supporting Creating Business firms/business intermediaries Infrastructure environment Private Sector Development Interventions PSD Interventions Supporting firms/SMEs Economic activities do not stop during conflicts. Firms and SMEs provide supplies for people’s daily needs and are sources of jobs and income for people. Investment and assistance provided directly to firms/SMEs will help build and strengthen their financial and productive capacity. Supporting firms/SMEs through financial intermediaries Conflicts may adversely affect macroeconomic stability by distorting or breaking normal market operations and systems, such as the banking system. Supporting intermediaries such as the financial institutions or business services providers that offer either financial products or services to firms/SMEs will improve access to finance and facilitate access new markets which in turn help economic recovery and development. Creating infrastructure Conflicts and disasters may significantly damage or destroy major physical infrastructure, such as roads or power plants. This negatively impacts people’s day-to-day lives, limits access to basic services, and constrains economic activities. In addition, conflicts weaken the capacity of government agencies and the workforce. Assistance and support for rebuilding physical infrastructure, either through direct private sector investment or public- private partnerships, is critical for rebuilding efforts that address the basic needs of citizens and facilitate economic activity. Training and education are important for the local labor market, which in post-conflict situations includes many young, unskilled, and inexperienced ex-combatants. Improving business environments FCS countries have urgent needs to attract investments to create jobs and economic growth. Supporting governments in areas such as regulatory and tax reform helps create better business environments and removes barriers to doing business for both local enterprises and foreign investors. 7 Scope Box 1: Data gathering and canalization This systematic review is based on a desk review Each evaluation identified was grouped and cataloged of English language published evaluations of according to the following parameters: various investments and technical Interventions assistance/advisory services projects targeting the  Country of implementation and typology of FCS; private sector in FCS. The scope of the review  Types of interventions – investment or technical covers evaluations of interventions that occurred assistance (TA); project size, and donor/ company in FCS countries, based on the World Bank FCS involved; classification from 2006-15. It also includes  Sector/themes – supporting SMEs, finance, evaluations and studies that took place in infrastructure or investment climate;  Objectives of the projects/ programs with outputs countries that were not on the World Bank FCS and outcomes recorded; classification list, but experienced certain level of  Design factors which were either positive or conflicts – such as Rwanda, Uganda and negative; internal or external factors which might Colombia – to better understand how PSD have affected the projects/ programs; interventions have helped these countries recover  Whether the project was designed taking into from or minimize negative impacts of conflict account principles of peace/ stability/ conflict. events and subsequent fragile situations. Evaluations  Type of evaluation and evaluation methods used; Fifty-six evaluations were identified as eligible  Evaluator’s rankings regarding relevance, effectiveness and sustainability of projects; evaluations for the review according to criteria  Whether the project recorded outcomes related defined in the approach paper. Annex 3 describes to peace-building or stability; the review methodology and final set of eligible  Types of challenges faced by the evaluation. evaluations covered by the review. Annex 4 provides a bibliography of the evaluations reviewed. It is important to note that of the 56 independent evaluations identified, 15 covered projects in Iraq – more than one fourth of the projects. Of those, 14 were UNIDO projects, 13 of which focused on business advisory services to support livelihoods and SME development. At the onset of its work in Iraq, UNIDO decided to evaluate and learn from its pilot initiatives which would, ideally, improve implementation, once the results of the evaluations became available. The authors of this report find UNIDO’s approach to applying lessons learned during implementation of projects in unstable environments to be laudable, especially considering that many FCS interventions in the past 10 years were not evaluated by independent evaluators or made available to the public. In some cases, this approach helped improve projects after the pilot phase, while in others the benefits were limited. Nonetheless, this initiative provides an opportunity to learn about the effectiveness of interventions in Iraq. In reviewing each evaluation and synthesizing findings, we sought patterns related to business areas; investment, technical assistance (TA) or advisory services (AS) projects; and the principles of “do no harm” (DNH), conflict sensitivity and peace-building. We attempted to detail what types of projects were successful and why, according to multiple criteria. Where relevant, we included brief summaries of exceptional projects as short case examples for the reader. 8 Limitations The desk review methodology did not allow us to obtain clarifications through interviews, emails or informal discussions that may have helped interpret some results and impacts. For example, in many evaluation papers, the authors mentioned elements of project design or implementation which were vague or inadequate for proper project monitoring of outputs and outcomes. Sometimes the authors referred to or applied different terminology about outputs and outcomes. In these cases, it would have been helpful to interview project managers or agencies and companies which were involved in the projects’ implementation to fill these knowledge gaps. This could be an area for further research in future. 9 Key Findings Effectiveness and value of PSD interventions This section provides a broad overview of project effectiveness and factors that affected overall potential for success. Box 2: Effective PSD interventions in FCS In FCS, it is particularly important to have funding and local partners who are deeply committed to making the project work and produce results. Projects were most effective when both the local stakeholders and the investor or donor had an important stake in the project/program. Direct investments work best. Private investments supporting SME development or larger infrastructure projects, in partnership with local business people/ stakeholders, had the highest rating for effectiveness (75 percent). Creative approaches using local knowledge to support the most vulnerable provide high opportunities for success. Examples of this are the projects targeting at-risk populations (particularly conflict-affected youth) for SME development. These youth projects had an 86 percent success rate. Starting with small or pilot projects is an effective approach. FCS are by nature unstable with vulnerable populations frequently affected by war. Starting small and building on successes made it possible to demonstrate to local entrepreneurs and businesses that new initiatives can work, which in turn helped rebuild trust. UNIDO’s work in Iraq provided several examples of strategic pilot initiatives which had a longer-term focus on sustainability. Time must be built into projects to ensure capacity building can take place , thus leading to changes. A key element of effectiveness with improving business investment climate interventions is to ensure capacity building and 'hand-holding' take place to ensure that the public sector not only passes laws or reforms, but implements them in the manner they were intended - to improve the regulatory environment and help local businesses grow. Passing laws just for the sake of passing laws does not help significantly if the laws are not implemented and enforced and/or reformed procedures are not followed. The evaluations identified were reviewed according to the following four interventions areas: (i) projects supporting SME and entrepreneurs; (ii) Figure 1: Success by types of country category interventions supporting financial intermediaries to improve firms’ access to Successful Somewhat Successful Not Successful finance; (iii) creating infrastructure; and (iv) 80% projects that worked with governments to improve the business investment climate and 57% 60% 48% 47% enabling environment. Among the evaluation 43% 43% 43% 40% 32% 29% reports analyzed, 46 percent out of 56 projects 21% 29% or programs (or 26 projects) were rated by 20% 9% evaluators as mainly, mostly or very effective 0% 0% across all countries and types of fragility In Conflict Post Conflict Fragile Countries exhibited during the period of implementation. experiencing periodical conflicts 10 Figure 1 highlights the breakdown of successful PSD projects by country typologies: post- conflict, in conflict, fragile or graduated from the FCS list. Although the numbers of projects in each country category type are similar, it is noted that countries that experienced periodical conflicts and that were not on the FCS list fared better than those still on the FCS list. For example, Colombia, Rwanda and Uganda were 33 percent more likely to succeed in project implementation than fragile or conflict states. It follows by countries that were post-conflict (47 percent); and less than half of projects in conflict or fragile countries were successful (43 percent). Figure 2: Success by types of instruments Moreover, we categorized effectiveness ratings along the lines of whether the project was Successful Somewhat Successful Not Successful implemented based on a financial investment – 80% 75% either by a private company or an investment 60% branch of a development bank, such as IFC; 60% 44% 40% 40% technical assistance (TA)/Advisory Services 40% (AS); or a mix of both (Figure 2). The strongest 13% 13% 16% results occurred in projects with direct 20% investments – and by a large margin. Of 0% projects implemented through an investment, 75 Investment TA/AS Mix of TA/Grant percent were successful, as compared to 40 with investment percent for TA/AS and 60 percent where TA or a grant was mixed with an investment. Figure 3: Success by types of interventions We also reviewed success based on the type of intervention: supporting SME projects; improving access to finance (A2F); infrastructure (INF); and improving business investment climate (IC) projects. The most successful interventions were those providing direct support to SMEs, followed by projects that focused on improving infrastructure (Figure 3). What value did these PSD interventions generate? To answer this question, we looked at whether projects or programs were successful toward achieving both their short term results, such as number of people trained, and also their longer-term impacts, such as job creation or private investment. Of the evaluations covered by the review, we could only find 14 evaluations – some of which covered multiple projects – that reported on employment creation and seven which reported on business investments. Jobs or investments were not targets or indicators established in the design of the majority of projects. In total, an estimated 39,383 jobs were generated in 20 technical assistance and investment projects, and nearly $1 billion in investments was generated in over seven projects (one project covered initiatives in five countries). The average investment per project was $87 million. 11 An overview of job creation and investment generation is shown in the table below: Table 1. Overview of job creation and investment generation Intervention FCS country Instrument Jobs created Investment generation (number) ($ millions) IC Burkina Faso, Liberia, Rwanda, S. TA/AS 21,500 60 Leone & S. Sudan INF Uganda Investment 1000 170 SME Bangladesh, DRC, Liberia, Nepal Venture Not recorded in 280 Capital Fund evaluation report SME Colombia TA/AS 800 2 SME Iraq TA/AS 4959 Not recorded in evaluation report SME Kosovo TA/AS 2924 Not recorded in evaluation report SME Sierra Leone TA/AS 8200 Not recorded in evaluation report SME Sierra Leone Investment Not recorded in 440 evaluation report What creates success? When designing interventions in challenging FCS environments, one tends to seek answers to the same questions: Are there specific combinations of interventions which can predict success in FCS? Do certain interventions work better in conflict, post-conflict or other types of countries? For example, should we be implementing projects which support livelihoods and SME growth in countries in conflict, or IC in countries post-conflict? And ultimately, with which types of interventions should one start and how do we go on to the next steps? Project success and effectiveness can be influenced by design factors, such as stakeholder capacity; the presence of functional local markets and value chains; and whether there is sound management with the time, interest and knowledge of investors and/or local champions necessary to make a project work. Success can also depend on factors related to project implementation, such as political problems, withdrawal of financing by funders, expropriation, or health issues of implementing staff. In addition to these factors, a lack of security because of war, environmental catastrophes or politic instability are external factors in FCS that may affect the project implementation and ultimately affect the success of the intervention. Applying this framework, the review observed that internal implementation factors play a more important role - over external factors - in influencing project success. The interventions in all of the 26 projects rated as ‘successful’ were well-designed. A bad design, however, could lead to a high probability of failure – five of the failed projects were unsuccessful largely because of flaws in design. 12 What types of interventions were most likely to lead to desired results? Projects were largely successful in meeting the intended project outputs, with 84 percent of all 56 projects or programs evaluated meeting their targets for immediate results or outputs. Changes in behavior and longer-term sustainable results were harder to obtain, with an overall success rate in meeting a project’s outcomes (as considered by evaluators) to be 41 percent for all 56 evaluations. This low rate is not surprising when one considers the extraordinary challenges of working in countries which are either still at war, just recovering from conflict or where governments are unstable and fragile. In these countries, situations can change from one day to the next, and governments may rise or fall due to external and political factors which cannot be controlled by donors or investors. In some cases, however, the projects were not designed well given the local context, nor were outcome indicators well developed during the project design – factors which contribute to the low outcome rating. Eleven of the evaluations describe poor development of output and outcome indicators. Another three evaluations were ones carried out in the mid-term. Therefore, 14 of 56 (or 25 percent) of evaluations concluded they could not easily comment on effectiveness. These points will be discussed in the later section relating to the quality of the evaluations, and in the session of lessons and recommendations. In the following sections, we provide details of the reviewed evaluations along types of interventions, with regard to achieving the outputs and outcomes. We also describe the design factors that helped ensure success, with examples of what worked and what did not and describe the key indicators used to measure success. All evaluations were carried out shortly after (or at) completion, with no focus on impact. 1. Interventions supporting SMEs Interventions supporting SMEs classified in the review typically consist of technical assistance for micro or small businesses and training in a specific sector to expand or create new businesses. Twenty-six such projects were carried out in 11 countries. Half of these (13) were in Iraq. The 26 projects shared the goals of supporting the development of entrepreneurs, micro- enterprises and SMEs through provision of training in specific sectors, rehabilitation of businesses, and providing support to local chambers of commerce, business associations or cooperatives, with a large focus on rural agriculture and livelihood development. The main objectives of these interventions were to assist vulnerable and other populations recovering from war which would ultimately provide jobs, stability and economic development. Fourteen projects were implemented in countries in conflict (Afghanistan and Iraq); eight in countries post-conflict, one in a fragile country and one each in the countries of Colombia, Rwanda and Uganda. The breakdown of SME support projects in terms of achieving outputs, outcomes and overall effectiveness is as follows:  Projects reaching output targets: 21/26 (81 percent);  Projects reaching outcomes targets: 12/26 (46 percent); 13  SME projects considered effective by evaluators: 15/26 (58 percent). SME support projects were reported to achieve their outputs as measured by:  People trained in more efficient farming methods, light manufacturing, or services;  Training of trainers;  Establishment of local business associations and training centers;  Development of training systems (also in schools);  Training of vulnerable populations—particularly youth at risk of returning to conflict, women;  Refurbishment of plants or factories. Outcome measures linked to sustainability were measured by:  Jobs created;  Investments generated;  Deployment of new production methods: farming, agro-industry, other manufacturing;  Increase in income and/or savings and decrease in poverty, in percentages;  Community development plans created;  Improved standards implemented for production. What design factors contributed to success? For the purpose of this meta-evaluation, it is important to remember that half of the projects evaluated in this sample were UNIDO projects implemented in Iraq. And all but two of the 13 focused on training and job creation during a time of great instability. The success factors related to designing the work in Iraq with local entrepreneurs and SMEs were:  Providing training that suited local needs to ensure that training skills were marketable, met the needs of the local market, and provided support to local supply chains;  Building on pilot programs to keep elements which work and build on lessons learned;  Winning the confidence of local entrepreneurs that the project would help support their livelihoods and overall survival, as opposed to using a top-down approach whereby the design elements factored in would not meet local needs or capabilities;  Establishing a local steering committee that would “champion” the projects. This was necessary in Iraq during a time when UNIDO staff could not be based in country due to security reasons, forcing them to manage the project remotely;  Factoring in sustainability – new training centers and rebuilt industry must be sustainable to contribute to long-term economic growth. For projects outside of Iraq, design factors necessary for effectiveness of the projects were nearly identical, even though the FCS circumstances were different, for example:  Developing creative projects with the local community that meet its needs, such as two 14 Youth Employment projects in Sierra Leone. One was implemented by the German international cooperation agency (GIZ). The second was a United Nations Development Program (UNDP) implemented by the International Rescue Committee (IRC). See details in the story presented in Text Box 9;  Building on pilot programs – designing and communicating a monitoring system up-front that would track initiatives (which may or may not work) to build on lessons learned. This was the case with a USAID-funded village savings and loan project that focused on income growth for vulnerable populations and youth safety/health in Makamba and Bujumbura provinces of Burundi;  Factoring in sustainability with a holistic approach to business growth – the specific needs for skills in specific sectors where there is interest and potential for growth must be considered. This approach was part of IFC’s design for the successful Kosovo Private Enterprise Program;  Taking a holistic approach to consider financing for entrepreneurs and SMEs, as well as developing skills for pricing, sales, marketing and distribution;  Designing projects with external investment or investment financing to help ensure sustainability. An example of a highly successful European Union (EU) intervention in Sierra Leone highlights a combination of the above factors that lead to the project’s potential for future impact. Box 3: Sorghum in Sierra Leone—an EU project The project focused on the value chain from producer to market development, and pricing to buyers. Components of the project were to: (i) Identify and introduce high-yielding sorghum varieties with high- quality industrial processing characteristics; (ii) Establish rapid-multiplication farms and sorghum collection centers; (iii) Form and train village-level producer associations and credit groups providing services and financing that enhance the ability of farmers to market increased amounts of quality sorghum; (iv) Assist reliable producer groups and leading commercial farmers in entering into longer-term partnerships with the beverage industry, which ultimately sought to multiply improved sorghum varieties for the next growing season; (v) Train private sector agribusiness dealers in supplying inputs to sorghum farmers through market mechanisms. Was the status of conflict important for a project’s success? In general, SME projects fared better after the conflicts had ended (75 percent and 67 percent success ratings for post-conflict and fragile countries respectively). However, the fact that 44 percent of projects were able to reach their outcomes while the country was in conflict, and in high-tension areas is impressive. The success ratings are largely due to the design factors noted above. Two out of three projects in “graduated countries” were rated as being effective and successful in reaching their outcomes. Both of the successful examples of projects or programs in Colombia and Uganda were based on external financing or investment. They relate to projects in countries which have “graduated” from being on the fragile or conflict-affected countries list: 15 Box 4: External financing and local support leads to success in Colombia and Uganda (i) Max Havellar invested in Fairtrade bananas in Colombia, a project which had strong leadership, management coordination with local stakeholder needs, and a holistic approach on how to improve the entire supply chain. Nearly 96 percent of 440 local small holders surveyed in banana production stated that their lives and financial conditions greatly improved as a result of the project, which was designed to help farmers increase production and advance economically. (ii) The Norwegian Government (GON) was assessed for its engagement over 15 years in Uganda: a longer-term approach to investments and SME and business development projects in Uganda. Despite the fact that Norway is not a key trading nation with Uganda, investments were targeted to Ugandan needs where there were overlaps with Norwegian interests, such as in environmental projects. Over the years, Norway invested in microcredit banks in Uganda; a re-forestation project which employed at least 800 people; and investment in a power plant as well as waste management services. This longer term investment approach has proven effective. Poor internal and external design factors, including a lack of funding or political influences, ensure that projects do not succeed and are not effective. Projects which were not successful were designed with little or no planning for sustainability. Local stakeholders, including vulnerable populations such as internally-displaced persons (IDPs), war widows and others, were not asked what kind of training they wanted or could sustain. Also, projects were possibly chosen based on local politics and unrealistic timeframes rather than fulfilling real needs. An example of this was a pilot project to rehabilitate a single dairy plant in Iraq within a very short timeframe of 16 months. The pilot focused on only one plant rather than the whole sector in the area at the request of local partners in country. Other examples of projects which were not successful had internal design flaws and external funding obstacles. Two UNIDO projects in Sierra Leone and Eritrea had the same objectives – to strengthen SMEs and local entrepreneurs through capacity building and national-level policy development. The project in Sierra Leone did not obtain the funding to proceed, while the one in Eritrea had poorly-designed goals and outcomes, making it difficult to assess progress made under less-than-optimal planning conditions. 2. Support SMEs/firms through supporting financial intermediaries —Access to Finance The 10 project evaluations covered, amongst others: microcredit banks in Afghanistan and Haiti; a country-wide analysis of microcredit banking in Afghanistan; venture capital funds for countries in Africa and Asia; credit guarantees for coffee growers in Rwanda and the earthquake- affected communities in Haiti (both USAID projects); revival of credit unions in the Solomon Islands and financial services for underserved populations in the Pacific Region. The breakdown of A2F projects in terms of achieving outputs, outcomes and effectiveness is as follows:  Projects reaching output targets: 6/10 (60 percent);  Projects reaching outcomes targets: 3/10 (30 percent); 16  A2F projects considered effective by evaluators: 3/10 (30 percent). Output results were measured by:  Financial Sector employees trained;  New banking services developed. Outcomes were measured by:  Loans provided (typically by microcredit banks);  Volume of lending;  Growth rate of banks;  Number of new branches opened;  New investments generated;  New banking services being used. What design factors contributed to success? There is much overlap in design factors across all business areas, and Access to Finance is no exception. Below are some critical design elements that supported success for three successful A2F projects implemented by IFC. The overlap with successful design factors in SME support projects is clear:  Investments in financial institutions (banks) are critical for the success of the project. As described in the text box below, all end-of-project evaluations of banks in which IFC had invested showed positive outcomes (Afghanistan, Haiti and the SME Venture Fund);  Technical assistance in training bank staff and/or bank clientele in new banking methods or the use and importance of new instruments is critical;  Knowledge of local market needs is critical for overall effectiveness and reaching the project’s objectives. Project implementers should know whether microcredits or other small loans are contributing to business development or supporting consumer purchases (e.g. cars, food, clothing, wedding expenses, etc.);  Pilot project testing followed by expansion of successes works best. Design the investment project with the aim of scaling up and expanding the potential for other bank branches or investment projects in future, based on the success of the initial investment or pilot approach. Globally speaking, IFC has been more successful than other donors in carrying out financial sector projects, with three out of six projects rated as effective and two out of six considered to be too early to rank, since these evaluations were carried out prior to project closure. Three of these projects are described below. The status of the country as an FCS did not seem to play any role in the success of a project’s implementation. 17 Box 5: IFC’s Banking investments in Afghanistan, Africa/Asia and Haiti The First Micro-Finance Project in Afghanistan has been evaluated very positively. One of the key factors for success was that IFC was an investor in this bank, and provided technical support – it was therefore fully committed to getting results and seeing its investment work. In addition, IFC hired consultants who knew Afghanistan and the local market well to provide advisory services. In Africa and Asia, IFC established the SME Ventures Investment Project covering the Democratic Republic of Congo (DRC), Nepal, Liberia and Bangladesh (which is not on the FCS list). This project was successful because (i) IFC invested its own funds in the venture capital fund and (ii) the project implementation team provided training for small companies via “Business Edge” and other IFC services. This project was also built on a previous pilot initiative in Africa, which provided valuable lessons learned. IFC funding of $100 million was instrumental in stimulating $280 million in additional financing for business growth in target countries. In a successful A2F project based in Haiti, IFC invested in Sogebank, the leading Haitian bank providing small businesses with microcredit. The project was successful because of IFC’s investment link with advisory services – training of Sogebank staff – in relationship to the expansion of microcredit and mobile banking services. Internal and external factors which hindered success included the following:  Underestimating local market needs;  Lack of accurate information about the conflict or fragility situation;  Failure to factor in closer monitoring or local monitoring of the project;  External events, such as the renewal of conflict in the Solomon Islands, forcing the premature closing of a project focused on establishing credit unions in the country.  Lack of an investment role by the donor or sponsor in banking projects. An evaluation of several microcredit schemes in Afghanistan carried out by the local research institute Afghan Research and Evaluation Unit (AREU) showed that many microfinance projects implemented by non-governmental organizations (NGOs) and others did not always support local business needs as intended, but rather paid for non-business expenses such as weddings, cars, and other consumer goods. When designing these projects, implementing NGOs did not take into account the needs of microcredit in rural areas or understand the potential conflict between business development and local or family needs. Another evaluation describes a project for the German-funded GIZ to help establish legal status for microfinance organizations in the Democratic Republic of Congo (DRC). The evaluation focused on one particular microfinance institution which appeared to have been poorly monitored because of insufficient coordination among other donors – even German-funded agencies – working in the DRC. 3. Supporting infrastructure development The seven projects in six countries evaluated refer to:  A road development project to connect rural populations in Kiribati;  A pipeline project between Chad and Cameroon; 18  Power for Kosovo;  Community infrastructure projects for vulnerable populations in Sudan;  Telecoms and power in Uganda; and  Renewable energy projects for SME development and improved energy in Rwanda. The objectives were typically to provide infrastructure to vulnerable populations that previously had little or no access to infrastructure services to help lift them out of poverty. Overall results of infrastructure projects are show below:  Projects reaching output targets: 6/7 (86 percent);  Projects reaching outcomes targets: 4/7 (57 percent);  INF projects considered effective by evaluators: 4/7 (57 percent). The FCS status of the country does not seem to have played a role in effectiveness, with two of the projects meeting the outcomes in conflict countries; one in a fragile location and one in a graduated country. On the other hand, all projects which succeeded had external investments associated with them. Results were measured as follows:  Environmental regulations passed related to infrastructure;  Reforms for privatizations;  Human rights issues addressed (in Sudan);  Replication of successful public-private partnerships (PPPs) in the same or other countries;  Power plants built (MW);  Power losses reduced;  Roads built (km);  Community development facilities built, such as schools, hospitals, drinking wells;  Pipeline built (km). What design factors contributed to success?  Working closely with the local government to champion the project and help develop plans, PPP contracts; policies for infrastructure;  As with the above, investments in these projects by the private sector (see the cases of two private companies in Sudan in the box below) or by donors;  Ensuring that infrastructure can help support economic growth and local business needs, such as the successful infrastructure project financed by a combination of investment and grant funding from the government of Japan (JICA) in Kiribati. This project concerned revitalizing distribution and transportation activities in South Tarawa by improving the road conditions in Betio, Bairiki and Bikenibeu districts and maintenance support. This JICA-funded project met a great need to improve roads for both personal use and business transport. The new roads resulted in reducing transport 19 times for fishing communities to bring their goods to market, which led to a large reduction in spoilage and improved product quality. By extension, this led to business and income growth for these agriculture communities. Box 6: Multinationals invest in infrastructure in conflict-ridden Sudan while supporting peace and stability Two infrastructure projects were carried out by two large international oil and gas companies in Sudan – Talisman and Schlumberger. Both decided to assist vulnerable local populations in areas where they were building or had pipelines, thereby helping communities which were at risk due to the war in Sudan. Talisman was involved with 21 local community projects around Khartoum – building schools, wells and a hospital. Talisman was concerned with sustainability and peace-building as well. Exceptionally, Talisman also became concerned about human rights abuses in the communities where it was working and hired two human rights specialists to work with them, coordinating with the NGO Human Rights Watch in Sudan and the UN High Commissioner for Human Rights. This example of Talisman proved that large international “rich companies” can and do act in FCS to prom ote well-being and peace in conflict-affected countries, while working to protect their financial interests as well. Schlumberger was also active in Sudan and worked in a suburb of Khartoum to build a school and drinking wells. 450 students could attend the school during the school year. The target population was internally displaced people (IDPs) from the Sudan-South Sudan war. The results were monitored on an ongoing basis and Schlumberger planned to improve the outcomes over time with ongoing monitoring built in post-completion for both health and education purposes. Of special note is an investment project in infrastructure in Uganda, where despite many delays, IFC was successful in its investment in the Bujagali power project and in the telecoms sector negotiating PPPs with the government. The government of Uganda used the telecoms investment model to negotiate subsequent deals in country. In addition, this became a model for infrastructure deals for other countries in the region for establishing their own power and telecom investment projects, including in the DRC, Sierra Leone as well as in other African countries. 4. Improving the business environment and investment climate (IC) Projects Thirteen projects were carried out in eight countries and two regions in African and Pacific countries. IC projects covered the following types of interventions: (i) public-private dialogue to enhance discussions across the public-private sector divide; (ii) drafting and passing legislation related to registration of informal businesses; (iii) trade regulations and improving customs controls; (iv) taxation; and (v) investment promotion. The objective of these projects was to improve local business regulations through reforms,which, if implemented and enforced, would ultimately facilitate business development and foster economic growth. The 13 investment climate projects were assessed as follows by evaluators:  Projects reaching output targets: 10/13 (77 percent);  Projects reaching outcomes targets: 4/13 (31 percent);  IC projects considered effective by evaluators: 4/13 (31 percent). 20 Results were measured by:  Reforms developed or enacted for improving the business environment;  Number of new businesses registered and assisted;  New offices or agencies established to support business;  Cost savings calculated for business with regard to reforms;  Implementation of reforms by government;  Improvement in “Doing Business” indicators. The majority of the projects related to investment climate or improving the business enabling environment have not been successful overall in their effectiveness. However, over three- quarters of the projects were successful in areas such as writing new business-enabling legislation, establishing public agencies for business support, or in getting businesses to register and join the formal business sector. Box 7: Main comments from evaluators related to design and unsuccessful project implementation  Lack of sound analysis of the capacity available for investment climate reforms to work;  Lack of inclusion of the local private sector in sounding out what reforms were most critical at the time and/or what kinds of technical assistance are required;  Poor analysis of the amount of time required – reforms for a government where many things are broken and need fixing take time. These reforms could not be achieved in 2-3 years;  Lack of sufficient government support for the investment climate reforms;  Failure of design to include sustainable measures such as enforcement of reforms. The primary reason for the lack of success was overly ambitious project designs, likely driven by donors’ desire to measure success in a very short period of time. Although legislative reforms typically take years to realize, many projects only had a 2-3 year time horizon. Expectations that these IC projects could succeed within a short period of time led to “Mission Impossible” situations. One exception was a successful project in Uzbekistan that took into account some of the important design factors critical for achieving results, as noted in the following text box. 21 Box 8: The UNDP in Uzbekistan: Establishment of a business forum and business arbitration for improving the overall business and investment climate Regarding the Investment Climate Project in Uzbekistan, UNDP had an ambitious set of objectives. These were: (i) facilitating dialogue with the government, business promotion institutions, private sector and other stakeholders; (ii) strengthening the capacity of the chamber of commerce/business associations and industry to better serve the needs of private sector, which would promote inclusive market development, also at the sub-regional level; (iii) creating a network of "Business Facilitation Centers;" and (iv) improving arbitration in Uzbekistan through raising awareness of the population and improving the quality of arbitration proceedings. The overarching reason for UNDP’s success was that the project was well -designed, with clear steps identified to carry out this work. UNDP had strong local partners and champions, particularly within local chambers of commerce. The project was also designed to take into account local entrepreneurial and business needs. Part of the technical assistance involved study tours and peer- to-peer learning in specific areas of interest, for example, in EU countries Italy and the Czech Republic as well as Malaysia. These involved relevant topics such as improving chambers of commerce and establishing an export association. Peer-to peer learning is often regarded as an extremely effective method of technical assistance as it provides face-to-face information sharing on how other, similarly-developing countries have been able to attain their goals. It is important to note that 81 percent of 550 SMEs surveyed at the end of the project were satisfied with the new services of the local chambers of commerce fostered by the technical assistance of the UNDP. Contribution of PSD interventions to peace and stability One common perception presented in the literature is that by supporting economic activity and generating jobs, PSD interventions can contribute to peace and stability. In this review, we identify and summarize the evidence regarding the PSD contribution to peace and stability from three angles of intervention design: (i) “do-no-harm” (DNH); (ii) conflict sensitivity; and (iii) peace-building. 1. The “Do-no-harm”, vulnerability and conflict sensitivity approaches and project success Twenty four projects reviewed had stated design objectives which specifically focused on or took into account: (i) vulnerability; (ii) DNH; or (iii) conflict sensitivity (or an aspiration to contribute to peace-building). These projects explicitly mentioned the notion of addressing vulnerable populations in the project design, for example, youth at risk, war widows, or IDPs. Some project designs alluded to peace and stability in describing the need for “pro-poor growth” and targeting remote or rural populations, addressing these as part of the project strategy. A few project objectives also focused on peacebuilding. Of the 24 projects, 14 were SME and business development projects; five were in investment climate; three in A2F and two in infrastructure (these relate to the projects in Sudan implemented by Schlumberger and Talisman described in the previous section). Both infrastructure projects were deemed effective by external evaluators. 22 Observation from these evaluations suggests that factoring the key principles of DNH, conflict analysis or peacebuilding measures in the project design may help enhance to the success of a project. Among the 24 projects, 15 (63 percent) were considered overall to be effective. These were broken down to the following business areas (See Table 2):  Peace-building SME projects: 9/14 rated effective (64 percent success rate);  A2F projects: 2/3 rated effective (67 percent success rate);  Infrastructure projects with peace-building elements: 2/2 worked well (100 percent success rate);  IC projects applying conflict sensitive approach: 2/5 rated effective (40 percent success rate). Six out of eight of the investment projects designed their projects with the principles of DNH, addressing vulnerabilities and with peace-building in mind; five of those were successful. The sixth was reviewed at mid-term and could not yet point to positive outcomes at the time. While we cannot attribute a project’s success solely to these design measures – incorporating knowledge of vulnerability and conflict to project implementation measures – we conclude that this type of design contributes to effectiveness. These projects are also better designed at the outset to meet local stakeholder needs. Table 2: Project effectiveness and design factors relating to vulnerability, conflict, peace and stability Types of interventions Support Improve Access Support Improve Direct SMEs to Finance infrastructure investment climate investment Success/ global effective 58% 30% 57% 31% 75% rating considering all 56 evaluations Success/ effectiveness rating 64% 67% 100% 40% 83% of projects where peace- building measures factored in the design (24 projects) 2. Contributing to peace and/or stability As noted above, less than half the projects specifically included notions of DNH or peacebuilding in the design of projects. And for those of the projects which did, only a few had indicators that attempted to measure how these design elements were achieved. Projects which did were mainly those directed to war-torn areas – particularly in Iraq – or to youth and women who had been negatively affected by war, or where there was a great concern of disenfranchised populations such as IDPs or returning child soldiers (Burundi, Sierra Leone, and Sudan). A few projects were also concerned with internal or cross-border trade and commerce in conflict areas or those suffering terrorist strikes or hit by natural disasters, such as in Haiti, Afghanistan, Colombia (FARC) DRC, and West Bank and Gaza. 23 Notwithstanding the above, 12 of 24 project evaluations (or 50 percent) made mention of what the project evaluators felt were contributions toward peace or assisting vulnerable populations. Given that these are relatively few, we mention the majority of these briefly below:  In Afghanistan, the UNDP Project for Peacebuilding and Economic Growth established 388 District Development Assemblies (similar to other donors’ Provincial Peace Councils) which created an organization for peaceful participatory government and local PSD planning accordingly;  In Burundi, the International Rescue Committee (IRC) implemented a project under the USAID Displaced Children and Orphans Fund (DCOF) for village savings and loans for vulnerable populations, also succeeding in their goal of improving the lives of vulnerable children who were beaten or abused by their families;  In the DRC, multiple donors focused on peace-building initiatives, some of which involved artisanal mining in eastern DRC. These initiatives resulted in increased cooperation, transparency and dialogue for economic development with cross-border trade with Rwanda. Donors’ efforts also contributed to reducing the risk of future regional conflicts;  In Iraq, four of the 14 UNIDO SME projects implemented there tracked employment results with regard to youth-at-risk; the re-integration of IDPs; returning soldiers and war-affected women. While one would have to return to these projects to see the longer term impacts related to job creation, at least a handful of them appear to have positively affected vulnerable groups;  In Sierra Leone, both GIZ and UNDP focused on youth at risk. The GIZ project was a traditional one with job training and support that would promote employment and SMEs growth. In this project, twice as many youths were trained as was originally envisioned with many jobs created. The UNDP creative project for youth is described in detail below. 24 Box 9: UNDP in Sierra Leone and its creative micro-franchising project for youth-at risk The UNDP micro-franchising pilot, implemented by the IRC, tested a new model of youth training and franchisor-franchisee relationship brokering that built on locally available assets and capabilities. It sought to align incentives among youth, businesses, funders, financial institutions, and the youth-serving community. The role IRC played was the following: it served as an intermediary actor, and relationship broker between businesses that were not typically already franchisors – such as bakeries, and the provision of bakery deliver services – and potential youth franchisees. IRC identified and provided youth with training, equipment, and business skills to establish and sustain a viable franchisee business. IRC also identified businesses that would benefit from franchise distribution of their products or services. IRC catalyzed a business relationship between youth and business based on franchising principles. This pilot project was deemed to be very effective in using a non-traditional approach to engage and employ youth, also using innovative methods of peer-to-peer support between young people engaged in the project. It provided jobs at the outset instead of training only. The project also has (to date) had the success of keeping youth engaged, feeling more positive about themselves, and out of street fighting or just hanging around on streets, becoming more integrated into the community through their businesses developed so far. In addition to supporting employment at the individual level, a number of project stakeholders felt that the micro-franchising project played an important role in strengthening private sector development in Sierra Leone as a whole by expanding markets for local SME owners.  In South Sudan, as noted in earlier sections, both Talisman and Schlumberger were instrumental in supporting war-affected populations with infrastructure that would improve their lives in terms of health and education. This contributed to stability for people affected by human rights abuses and who were uprooted during the Sudanese war.  In Colombia, Max Havellar’s Fair Trade project – highlighted in one of the SME cases in the earlier section – was instrumental in targeting areas which were traditionally harassed or affected by the terrorist group FARC. Its work in economic and farming development provided an expanded market and buyers who would help them move out from under FARC control. Quality of the evaluations As indicated in the earlier section, one quarter of projects were judged by evaluators as difficult to assess for two reasons. Firstly, many projects lacked appropriate designs with regard to the availability of baseline information, established logical frameworks and/or clarity about results and outcomes that projects aspired to achieve. Secondly, a few project evaluations carried out were mid-term reviews, where it was too early to ascertain whether the projects would be effective in meeting the planned objectives. Therefore, in 25 percent of the evaluations, evaluation authors were undecided in their conclusions and comments on project/program effectiveness. The “evaluability” of the projects was compromised in these instances. Related to the above remarks, and due to the nature of the context of the FCS countries in which projects were carried out, the overwhelming majority of evaluations (54 out of 56) were non- 25 experimental design, using mainly qualitative methods. The design, methodologies and approaches are summarized below. Design. Fifty-one of the evaluations were post-test only, non-experimental design. Three of these were conducted at mid-term. Of the remaining evaluations, one was conducted with random control trials (RCT) and one was a pre-post evaluation. The project designed at the outset to be evaluated via RCT was one of the most successful, as described below via the evaluation of “Urwaruka Rushasha,” managed by the IRC: this was a randomized impact evaluation of village savings and loans associations and family-based interventions in Burundi. Box 10: Randomized Control Trial - Evaluation of Urwaruka Rushasha in Burundi The USAID-funded Urwaruka Rushasha project was designed to address risks facing children in the Makamba and Bujumbura rural provinces of Burundi while filling gaps in evidence around i) the impact of Village Savings and Loans (VSLA) interventions on household economic outcomes and child well-being; and ii) the potential for parenting programs to improve child protection and well-being in poverty and conflict affected communities. The project was designed at the outset with a treatment and control group. The design included testing, monitoring, and close communications about the program with the engagement of local stakeholders in the trial. This buy-in was important. The project design also incorporated vulnerability of the families and children in post-conflict Burundi. Because the evaluation was planned from the outset, with strong stakeholder engagement, it was much easier to measure the strong project outcome benefits against the control group. Methodology. The methodology of 15 of the evaluations was purely qualitative, involving a desk review of project reports and other materials, with interviews, conducted in country of a sample of project beneficiaries and other key stakeholders. In 39 of the evaluations, mixed methods were used, typically combining a survey of beneficiaries with one-on-one interviews and a desk review of the project reports and other documentation. Conflict Sensitive Approach. Twenty-five percent of evaluators specifically mention a conflict-sensitive approach to the evaluation. Not surprisingly, this was because 13 out of 14 evaluations were carried out in countries in conflict during the time of evaluation. In many other evaluations there was no mention of conflict sensitivity; the evaluators themselves were often affected by security or conflict issues, particularly in attempting to carry out evaluations in Iraq. Key challenges and constraints in conducting evaluations The review identifies several challenges in carrying out a comprehensive, high-quality evaluation. Some of these are not surprising given that these evaluations were carried out in FCS, including many still in conflict. Forty-seven (84 percent) of 56 evaluations/reviews discussed the issues and challenges that the evaluators encountered when carrying out the evaluation. The most notable challenge was “poor or no data by the evaluators. This was mainly because of poor 26 design quality related to logical frames and unavailable data for measuring outcomes and effectiveness. “The review team had to face the important challenge of not having access to reliable data due to a weak monitoring system, the poor quality of indicators and the lack of compilation of financial figures” (GTZ, 2009). Likewise, as noted by the evaluator of IFC’s program Removing Administrative Barriers to Investment Project – Phase III - Sierra Leone: “Data obtained from companies and professionals are often at odds, and widely variable… This means that calculations are inevitably based on rough estimates. In addition to this, coherent data on labor costs are also difficult to gauge given the huge differences in wage levels across the various types of businesses.” Moreover, the lack of institutional memory of about the program/project was another challenge that significantly limited the quantity and quality of data available for the evaluation (United States Agency for International Development, 2009). Thirteen evaluations mentioned security as an issue. Fighting and insurgencies limited where evaluators could travel when reviewing some projects in Iraq; sometimes local counterparts had to carry out interviews or surveys because evaluators were not permitted to travel in-country. In some instances – particularly in Iraq – the evaluators noted that biases were introduced when the external evaluators had to rely on local counterparts who selected interviewees. Surveys were sometimes influenced by local implementing agents or bodies. While a sampling strategy may have been in place to randomize surveys and interviews, this may have not been feasible due to security issues. A lack of baselines was mentioned in eight out of 56 evaluations. In the absence of baselines, it was difficult to draw reliable conclusions about the effectiveness of the interventions. Finally, 10 evaluation teams noted that there was insufficient time to carry out the assessment of the effectiveness and outcomes/impacts of the programs. A few programs have raised attribution issues in discussion of the evaluation findings. It is important to note that these challenges were not the only obstacles for designing and applying robust approaches to evaluations; however, they affected the design of the interventions. In many instances, the evaluations had to compromise the design and methods to accommodate the reality of “evaluability” in terms of missing the underlying theory/assumption for the interventions, well defined indicators and missing information and data. Conclusions and Implications The objective of this review is to identify and summarize evidence on the effectiveness of various PSD interventions implemented in FCS, a topic that is drawing increasing attention from international organizations and donor communities. 27 This review aims to answer a central question: what is the effectiveness of PSD interventions in FCS? Findings suggest that overall, PSD interventions in FCS have positive effects.  The majority of PSD projects (84 percent) were effective in achieving their planned results; nearly half were also successful in reaching the planned outcomes and beyond in terms of job creation and investment generation;  Of the four business areas programs analyzed – SME support, Access to Finance, Infrastructure Projects and Investment Climate reforms – the most effective projects were those supporting SMEs. This was followed by Infrastructure Projects, then Access to Finance. Investment Climate had a relatively low success rate overall;  Successful SME projects involved training of entrepreneurs for livelihoods development and/or rehabilitation of micro or small businesses, including providing access to markets for the businesses supported;  Direct investments or larger infrastructure projects, in partnership with local business people or stakeholders, showed the highest rating for effectiveness. Because of the small number of available evaluations and the quality of the many evaluations identified, this review was unable to draw definitive conclusions on the effectiveness of different types of instruments for effectiveness in specific types of FCS countries;  Although there was no direct evidence on the contribution of PSD interventions to peace and stability, of the 24 projects incorporating DNH and/or peace-building in the design phase, 63 percent were successful according to the evaluators. Of the 32 projects which did not incorporate elements of peace-building into the design, only 34 percent were effective. This suggests that conducting conflict and vulnerability sensitivity analysis and factoring in peace- building or the do-no-harm concept in project design appears to contribute to the effectiveness and positive results of projects or programs;  Of 19 projects which incorporated positive design criteria, 89 percent were successful. One of the main lessons learned regarding project design – instrumental to project success – was the need for close engagement with local communities and stakeholders to ensure that projects address the needs of local communities.  The review noted several recurring factors related to project failure: (i) poor project design; (ii) lack of investment; and (iii) and insufficient engagement of the private sector, which was considered instrumental in achieving both the planned outputs and the project outcomes; The review revealed that most evaluations experienced challenges such as a lack of baselines, poor data and security concerns; in many cases these challenges affected the comprehensiveness of the evaluations, thus raising questions about the robustness of conclusions drawn. Another important finding was that poor quality evaluations were, in some cases, not due to FCS-related or external factors; rather, the cause was a failure to follow program logic or theory of change in designing the project, thus making sound evaluation impossible. One evaluation was conducted using an RTC design for a USAID project in Burundi. Evidence from this evaluation points to one of the most successful of all the projects: the evaluation was fully integrated into the project design and operations, whereby local communities were well informed at the project design stage of the reasons for control and treatment groups. Local 28 stakeholders were informed of project monitoring from the start and were engaged throughout project implementation. Implications The review findings have implications for institutions, program leaders, evaluation researchers and practitioners. To be successful in FCS programs involving PSD interventions, institutions and organizations should consider selecting the appropriate instrument and identifying sectors that show higher potential for greater development impact. It is critical to provide financial support for countries to cope with or recover from conflict. At the program and project level, the program managers should consider;  Applying the appropriate instrument and focusing on sectors that show higher potential for greater development results;  Engaging local communities, including vulnerable and at-risk populations, to develop creative solutions;  Starting with small, pilot projects to build on successes across the country or sector;  Analyzing and factoring in fragility and conflict as part of project design work, particularly in assisting vulnerable populations who are affected by the conflict;  Engaging local ‘champions’ or steering committees throughout implementation to support monitoring project progress and steer them towards success; and  Finally, addressing sustainability from the outset and ensuring that plans are achievable and realistic within a short period of time – particularly for projects of 2-3 years. The evaluation community should establish more evidence about the effectiveness of PSD interventions to improve our understanding of their value and potential contribution to peace and stability. Currently, few rigorous evaluations of PSD interventions in the FCS space are available. The evaluation of USAID project in Burundi using RCT has set a good example demonstrating that rigorous evaluation in FCS is possible. While some FCS-related factors, such as security, may be inevitable when carrying out evaluations in FCS settings, many important factors can be addressed, including engaging the project team earlier and working with it to ensure the program/project log frame is well- established and indicators are clearly defined. It is also useful to involve the local communities and/or beneficiaries in the design of M&E systems. When M&E is a central part of project planning, it improves longer-term sustainability and effectiveness of an intervention. When people and communities are engaged in active monitoring of projects, action can be taken when results go off track. Stakeholders will also be more engaged in ensuring the sustainability of a project. Finally, there is a great need for experimenting and adopting new and innovative approaches to evaluation, for example, by using technology to address challenges faced in the FCS setting. 29 Annexes 30 Annex 1. Country typology In-Conflict. These are countries where civil and/or tribal war or conflict has been present during project implementation, and where conflict affects the way the government is able to provide services to the population country-wide. The government has likely been in transition (there is no functioning Government or body to pass laws) during implementation or has proved reluctant to carry out reforms and provide civil services, or unable to do so, due to instability, corruption or systems which are not transparent. Five countries in this group are: Afghanistan Sudan Democratic Republic of Congo (DRC) West Bank and Gaza Iraq Post Conflict. These are countries which, at the time of project implementation, were coming out of a recent conflict that ended in the past decade, and where – despite political instability or weak government capacity – there has been visible interest on the part of the government to move ahead with regulatory reforms. Nine countries/ regions in this category covered in this review are: Burundi Nepal Eritrea Pacific Islands (Grouped as: Papua New Guinea or Georgia PNG, Timor-Leste, Solomon Islands) Kosovo Sierra Leone Liberia Solomon Islands (2005-13) Fragile. This is a cluster of countries that, similar to the above “conflict” category, have governments that cannot or will not deliver core functions to the majority of its people, due to internal governance problems or external factors, including climate change or related environmental difficulties. There are many definitions of “fragile states” and generally speaking, governments of these states lack the will and/or the capacity to manage public resources, and support poor and vulnerable groups (DFID, 2005). There is, however, no universally accepted classification of ‘fragile states.’ Five countries in this group are, with Solomon Islands also post-conflict: Haiti Tajikistan Kiribati Uzbekistan Solomon Islands (2004-05) Countries experiencing periodical or temporary conflicts. These are a group of countries that had been experiencing periodic or temporary conflicts however, as of 2005 were no longer considered by the World Bank Group as either fragile, in-conflict or post conflict. In this review, we chose the following three countries for closer examination: Colombia, Uganda and Rwanda. 31 Annex 2. Definitions—effectiveness and evaluability In reading the 56 evaluations covered by this review, we defined effectiveness as how external evaluators measured the degree of success of the project managers in being able to attain the planned results and objectives of a project. In many cases (25 percent of projects), outcomes were poorly defined or there were no outcomes predicted. In addition, project designs sometimes confused outputs and outcomes in the project documentation though short-and long-term results were achieved. This affected the “evaluability” of the projects, or, the ability of evaluators to determine how well projects were implemented, since the basis for measuring success was not always clear. Therefore, in some cases the most evaluators could do was assess projects as ‘mainly’ or ‘mostly’ effective. Below we provide an explanation for how we measured degrees of success related to project effectiveness in the evaluations: (i) Projects were categorized as ‘successful’ with regard to effectiveness, related to the evaluations carried out, when evaluators concluded the projects were ‘fully,’ ‘mainly’ or ‘mostly’ successful at project completion, according to the following criteria:  Both outputs and outcomes were reached as had been planned at the design stage and often exceeded expectations, with strong prospects for sustainability in future;  Pilot projects were considered successful and recommended for replication;  Outputs and the majority of outcomes were reached, although some outcome indicators were determined by evaluators to have been overly-ambitious;  Outputs and outcomes were reached and evaluators considered that it was likely that project outcomes would be sustainable, though it was too early to assess shortly after the project’s completion. (ii) Projects were categorized as ‘somewhat successful’ given the following:  The projects were far-reaching (even cross-border) and some stakeholders had benefitted from an effective project, whereas others did not (example: the Cameroon- Chad pipeline where Cameroon had many positive socio-economic outcomes, whereas Chad used project proceeds for purchase of weapons to fuel the conflict);  When there were many positive outputs, but outcome indicators were not defined or there were few positive outcomes (example: the project in Iraq to support improved date production. In this project, no outcome indicators were defined yet the project attained very good results);  When many positive outputs (such as training) were achieved, but with long project interruptions or with delays, making it difficult to know whether outcomes would be sustainable;  When the evaluation covered multiple interventions over several years where some were positive and others not (example: USAID’s several interventions in the energy sector in Kosovo – some worked but others were not considered effective). 32 (iii) Projects were categorized as ‘not successful’ regarding effectiveness when:  Outcomes which were planned were not attained, such as with a lack of endorsement of new reforms or policies or with establishment of new systems which did not seem sustainable at completion due to lack of capacity;  Goals were not reached due to lack of time or resources;  Activities were carried out with positive outputs, yet the logic was flawed, since ultimate stakeholders and/or end beneficiaries did not benefit as intended (example: the Australian-funded Regional Assistance Mission to the Solomon Islands “RAMSI,” where reforms were implemented but not considered key for private sector development (PSD) for local businesses at the time); or  At mid-term the project was evaluated and – at the time – did not show progress toward achieving the stated objectives/ outcomes. 33 Annex 3. Review methodology I. How did we collect the universe of evaluations included in the final list? The review team took several process and steps to identify eligible Summary of Review Process and Steps evaluations. First, we carried out Google and IFC/ WB data searches based on studies/ or evaluations of investment or technical assistance projects or programs carried out in countries during 2006-14 that appeared on the World Bank Group’s (WBG) FCS list during that period. For each of the 55 countries on the FCS list over that timeframe, we conducted six word/ phrase searches for English language documents: 1) country, "private sector development”, evaluation 2) country, "private sector development”, report 3) country, "private sector development”, study 4) country "private sector," investment, evaluation 5) country "private sector," investment, report 6) country "private sector," investment, study We also added evaluations from sources who provided reports on private companies and their “Peace- Building” or “Do no Harm” (DNH) interventions in a few FCS. Finally, to the original list of 55 countries in FCS, we searched for evaluations in an additional three countries that had been post-conflict for some time and, as of 2006, were no longer on the WBG FCS list: Colombia, Rwanda and Uganda, using the same search phrases listed above. We added these in order to determine whether there was anything noteworthy done in these countries which was more or less successful in the interventions – to provide lessons for the countries which are emerging from FCS. This gave us a starting point of 312 evaluations/ reviews and studies. II. How did we finalize the number of 56 Evaluations? From this group of over 300 documents, we took out all evaluations which were self-evaluations, including those assessments which were made by a department within an institution, such as the Independent Evaluation Group (IEG) within the WBG or the Independent Development Evaluation (IDEV) of the African Development Bank: this left us with 140 independent evaluations. We removed those documents covering multiple countries, such as evaluations of the Mekong River area, which included Thailand or other countries not on the FCS list. We also omitted evaluations which covered multiple donor initiatives, or multiple sectors, with little focus on PSD, as well as evaluations which were listed as such but were in fact case studies or research articles. We further removed evaluations which covered projects that were implemented in countries which at the time of project implementation were not on the FCS list. For example, this removed evaluations of Cambodia, Nigeria, Laos and Tonga. While these countries had been on the FCS during a few of the years from 2006-13, at the time of implementation, the countries were not considered FCS. This brought us to the final number of 56 evaluations. 34 Annex 4. A list of evaluations reviewed Afghanistan 1. “From Access to Impact: Microcredit and Rural Livelihoods in Afghanistan;” Afghan Research and Evaluation Unit (AREU); Paul Kantor; June, 2009; Afghanistan. 2. “Evaluation Report of the First Microfinance Bank of Afghanistan;” Banyan Global; Mayada El - Zoghbi; August, 2008; New York. 3. “Evaluation of CPD Outcome 6: Diversified Livelihoods, Private Sector Development and Public-Private Partnership;” Abhijit Bhattacharjee et al; November, 2013. Africa reports 4. “Evaluation of the World Bank Group’s Investment Climate Programs: Focus on Impact and Sustainability - Burkina Faso, Liberia, Rwanda, Sierra Leone and South Sudan;” Economisti Associati; November, 2014; Bologna. 5. “A Randomized Impact Evaluation of Village Savings and Loans Associations and Family-Based Interventions in Burundi”; USAID; Jeannie Annan et al; March, 2013. 6. “Evaluation of the Demonstration Effect of IFC's Involvement of Infrastructure in Africa”; (Cameroon - DRC pipeline case) Castalia Strategic Advisory; February, 2011. 7. “Promotion of the Private Sector and Civil Society - Focus on Micro-Finance Component: DRC” (GTZ); Adrian Marti et al; July, 2009; Eschborn. 8. “Evaluation of SME Ventures: Final Report. Project No. 11-16058 (IFC);” Bella Research Group; Dr. Josh Lerner et al; April, 2014. 9. “The Impact of Mining in the Democratic Republic of Congo - Performance to date and future challenges;” Oxford Policy Management and Synergy Global; October, 2013. 10. “UNIDO Integrated Program for Eritrea: Sustainable and Competitive Industrial Development;” Leny van Oyen et al; December, 2005. 11. “Investment Climate in Africa Program – Four Country Impact Assessment: Liberia Country Report”; Economisti Associati et al; March, 2011. 12. “Outcome Evaluation - Inclusive Private Sector Development - the United Nations Development Programme (UNDP) in Sierra Leone”; Dr. Stephen Chipika et al; October, 2012. 13. Employment Promotion Programme (EEP) Sierra Leone – Interim Evaluation for GIZ ; Dr. Hildegard Schuerings et al ; May, 2011 ; Frankfurt and Freetown. 14. “Sierra Leone UNIDO Integrated Support Programme - Post Conflict SME Support Programme for Industrial Development and Poverty Alleviation;” Margareta de Goys, et al; 2008, Vienna. 15. “Investment Climate in Africa Program – Four Country Impact Assessment: Sierra Leone Country Report; Economisti Associati et al; March, 2011. 16. “International Finance Corporation (IFC) Tourism Investment Generation Program in Sierra Leone”; Mike Fabricius et al; June, 2012. 17. “Sorghum Supply Chain Sierra Leone – A Sustainable Project ?;” Stenden University; Anne Deters; February, 2011; Amsterdam. 35 18. “YouthWORKS Microfranchising Project Evaluation;” Schulich School of Business, York University; Sarah Murray et al; January, 2010, Canada. 19. “Impact Evaluation of Schlumberger’s Community Outreach in Sudan;” Channel Research; Emery Brusset; March, 2009; Ohain, Belgium. 20. “Evaluation of Talisman Energy’s Social Engagement in Sudan: 1998-2005;” Channel Research; Emery Brusset; January, 2006; Ohain, Belgium. 21. “Evaluation of Norwegian Business-Related Assistance: Uganda Case Study;” Devfin Advisers AB; May, 2010. 22. “Evaluation of the Demonstration Effect of IFC's Involvement of Infrastructure in Africa”; (Uganda Infrastructure case) Castalia Strategic Advisory; February, 2011. 23. “ Rwanda Bank of Kigali DCA Guarantee;” (a USAID project); SEGURA/IP3 Partners LLC; December, 2009. 24. “Independent UNIDO Country Evaluation – Rwanda;” Simon Taylor et al; 2011, Vienna. 25. “Investment Climate in Africa Program – Four Country Impact Assessment: Rwanda Country Report; Economisti Associati et al; March, 2011. 26. “Access to Energy in Rwanda: Impact Evaluation of Activities Supported by the Dutch Promoting Renewable Energy Programme;” German Rheinisch-Westfälisches Institut für Wirtschaftsforschung (RWI) in consortium with other research institutes in Europe; Jörg Peters et al; August, 2014. Iraq reports 27. “Independent Evaluation Report – Iraq - Promotion of Cottage Industries in Rural and Urban Areas Project;” Mr. Majid Slama et al; 2008, UNIDO, Vienna. 28. “Independent Evaluation Report – Iraq –Community Livelihoods and Micro-Industry Support Project in Rural and Urban Areas of Northern IRAQ (MISP II);”HAP Consultants; 2010, UNIDO, Vienna. 29. “Independent Evaluation Report – Iraq – Rehabilitation of the Date Palm Sector; 2011, UNIDO, Vienna. 30. “Independent Evaluation Report – Iraq – Enterprise Development and Investment Promotion (EDIP);” Ms. Henny Moeller Andersen, 2011; UNIDO, Vienna. 31. “Independent Evaluation Report – Iraq - Pilot Project for the Rehabilitation of the Dairy Sector in Iraq; HAP Consultants; Fredie Andersen et al; 2010, UNIDO, Vienna. 32. “Independent Evaluation Report – Iraq - Evaluation of Micro Enterprises for Reintegration of Internally Displaced Persons in This Qar Governorate;” Scanteam; David Gairdner et al; 2013, UNIDO, Vienna. 33. “Independent Evaluation Report – Iraq - Technology Acquisition to Restart and Generate Economic Transformation (TARGET);” David Gairdner et al; 2012; UNIDO, Vienna. 36 34. “Independent Evaluation Report – Iraq - Support for Job Creation and Self-Employment through Promotion of Micro Industries in Ninewa Governorate of Iraq (MISP IV);” Ms. Henny Andersen et al and SRD, Amman; 2011; UNIDO, Vienna. 35. Independent Evaluation Report – Iraq - Combined Independent Evaluation of: Enterprise Development through Information and Communication Technology (EDICT) and Investment Promotion for Iraq (IPI) – 2 Projects: FB/IRQ/09/007 and TE/IRQ/09/010; Ms. Ginger Cruz et al; 2013; UNIDO, Vienna. 36. “Independent Evaluation Report – Iraq - Job Creation through Cottage and Micro- Industries Promotion in Al-Qadessiya (MISP III);” HAP Consultants; 2011, UNIDO, Vienna. 37. “Independent Evaluation Report – Iraq - Promotion of micro industries for accelerated and sustainable livelihood recovery – Ninewa Governate of Iraq (MISP-V)”; David Gairdner et al; 2012, UNIDO, Vienna. 38. “Independent Evaluation Report – Iraq - Rehabilitation of the Mosul Dairy Plant;” 2012; UNIDO, Vienna. 39. “Independent Evaluation Report – Iraq - Enhancing Investments to Iraq through Industrial Zone Development (IZ);” Ginger Cruz et al; 2013; UNIDO, Vienna. 40. “Evaluation of Enabling policy framework for rapid economic recovery, inclusive and diversified growth and private sector development: Outcome 5, UNDP Iraq Country Programme Action Plan 2011-2014;” Sulaiman S. Wasty, David Gairdner et al; 2012. Other reports, in alphabetical order of country/region: 41. “An Evaluation of Fairtrade Impact on Smallholders and Workers in the Banana Sector in northern Colombia”; the Corporation for Rural Business Development (CODER); Carlos F. Ostertag et al; February, 2014. 42. “British Petroleum’s Baku-Tbilisi-Ceyhan and Southern Caucasus Pipeline Community Development Initiative (CDI4) Programme (Georgia);” GRM International; Emery Brusset; December, 2014. 43. “Small and Medium Size Enterprise Financing and Mobile Banking in Haiti (2 evaluations) - Evaluation of IFC MSME Sogebank and Sogebank MBanking;” Management Systems International; Jacques Bérard; October, 2012. 44. See above – combined evaluation. 45. “Evaluation of DCA Guarantees to Micro-Credit Capital and Sogesol, Haiti (A USAID project); Segura Consulting; August, 2013. 46. “MSME Sogebank;” IFC Haiti Technical Assistance project; Independent Evaluation Group; March, 2014. 47. “Ex-Post Evaluation of Japanese ODA Grant Aid Project: The Project for Improvement of Fisheries-related Roads in South Tarawa (Kiribati);” Ernst & Young Sustainability Company, Ltd; Keisuke Nishikawa; 2012. 37 48. Mid-Term Evaluation of the Kosovo Private Enterprise Program (KPEP – a USAID project); September, 2011 49. “Programmatic Impact Evaluation in the Energy Sector in Kosovo (USAID);” Mendez England and Associates; Arvid Kruze et al; September, 2012, Bethesda, Maryland. 50. “Review of IFC PEP-Pacific Final Report;” Nexus Associates, Inc.; April, 2010, Arlington, MA. 51. “Pacific Microfinance Initiative (PMI) - Mid-Term Evaluation Report;” Pete Power; June, 2013. 52. “An Independent Evaluation of the IFC/AusAid Solomon Islands Credit Union Advisory Services Project;” Wayne Thompson et al; July, 2008. 53. “The Private Sector, the Solomon Islands and the Peace-Economic Dividend: Learning from the RAMSI Experience;” Kings College London; Anna Powles; December, 2013. 54. “Mid Term Review of the Rural Growth Programme in Tajikistan (DFID);” Coffey International Development; Graham Perrett et al; October, 2011. 55. “UNDP Project Business Forum of Uzbekistan - Evaluation Report;” Dr. Arkadi Toritsyn; October, 2010. 56. “Mid-Term Evaluation of Investment Climate Improvement Project: Fiscal and Policy Components (West Bank and Gaza; USAID);” David Dod et al; October, 2012. 38