82000 META-EVALUATION ON JOB CREATION EFFECTS OF PRIVATE SECTOR INTERVENTIONS Review of 39 evaluations conducted by IFC and other private- sector oriented development financial institutions (DFIs) that were carried out between 2000 and 2011. Gloria Paniagua and Anastasiya Denisova 9/4/2012 Contents Executive Summary ................................................................................................................... 3 Introduction ................................................................................................................................ 5 Identifying the Relevant Program Evaluations ........................................................................ 5 Central Questions Addressed ................................................................................................. 6 Challenges and limitations of the evaluation universe, data and methodology .................... 7 Employment effects by private sector interventions .................................................................... 8 Access to finance (A2F) ......................................................................................................... 8 Access to Infrastructure (A2I) ................................................................................................14 Investment Climate ................................................................................................................18 Training and skills development.............................................................................................22 Analysis of evaluation methodology ..........................................................................................27 Access to Finance..............................................................................................................27 Access to Infrastructure .....................................................................................................28 Investment Climate ............................................................................................................29 Training and skill development ...........................................................................................30 Conclusion and Recommendations ...........................................................................................32 Appendix 1: Methodology type by subject area .........................................................................34 Appendix 2: Summary of Evaluation Results by Employment Indicator .....................................35 APPENDIX 3: Results from IFC monitoring system data ...........................................................37 Results data from IFC‘s Development Outcome Tracking System (DOTS) ........................37 APPENDIX 4: Sources/ Databases ...........................................................................................38 REFERENCES ......................................................................................................................39 2 Executive Summary Though there is strong consensus that creating jobs is one of the key contributions of the private sector to poverty reduction, there is limited knowledge about which interventions, and in what country conditions, are most likely to catalyze job creation—and which activities are most beneficial to poor people. To address this knowledge gap, the International Financial Corporation (IFC) conducted this meta-evaluation or review of evaluations. The objective was to determine the employment effects of private sector interventions and the tools and combination of tools that can be used to estimate their effect on job creation, to help shape the strategies and operations of IFC and other private sector–oriented development financial institutions. Originally, 88 IFC and 59 external evaluations were identified. Evaluations not yet completed, public sector interventions, those that did not measure or estimate job creation, or cases where copies could not be obtained were excluded. After applying these criteria, 39 evaluations carried out from 2000-11 and commissioned by IFC and other organizations were included in the review. The findings are based on a synthesis of results from these evaluations, not all of which had the same objectives and methodologies. Methodological approaches ranged from quantitative analysis such as randomized control trials to focus group discussions and qualitative anecdotes from interviews. The evaluations covered Latin America, Asia, Africa, Eastern Europe, and the Middle East. Moreover, the meta-evaluation focused on the job creation effects of private sector interventions in four main areas: access to finance, access to infrastructure, the investment climate, and skills development and training. Main findings  Due to the diverse set of methodologies applied across different areas, the best approach to estimate the job creation effects of private sector interventions should be sector specific.  Only about 26.5% of evaluations (39 of 147 evaluations with an employment component extracted from databases of the World Bank Group and other relevant organizations) address job creation.  Improving access to finance can help firms expand their operations, which can have a positive effect on the quality and number of jobs created. The effects tend to be greatest for smaller firms. Combining access to finance with advisory services also tends to have a more positive effect on employment generation.  Improving access to finance for micro-enterprises can create jobs both through establishment of new businesses and through expansion of already existing ones. The latter effect tends to dominate in rural settings. Investments in services sector in urban areas and in agriculture in rural areas tend to create the most jobs. Collective loans are likely to have stronger effects on employment than individual ones.  Improving access to infrastructure has economy-wide positive effects on job creation. Net effects are positive but some sectors can experience job losses due to competition from imports after the construction of roads and ports. The predominant effects tend to be indirect.  Investment climate reforms—particularly business entry/registration reform and investment promotion—tend to lead to the creation of new firms and can have positive employment generation effects. But firms that enter after the reform may be less likely to survive their first two years than firms that enter prior to the reforms. So even though new jobs may be created, their duration and quality are not always clear. Nevertheless, even though some firms tend to exit the market after the reform, this has a potential 3 positive effect as it leads to a constant reallocation of resources/jobs to more efficient enterprises in the economy. Tax simplification reforms also lead to increase levels of registration (formalization) which, in turn, lead to much higher revenues, employment and profits among firms that register as a result of the program.  Training for youth produced either no effect or positive effects depending on the country and the way training was designed and delivered. Combining in-class training with on- the-job training tended to have positive effects. Training was the most beneficial for women and disadvantaged youth. 4 Introduction There is consensus among policy makers, donors, World Bank Group shareholders, private sector companies that creating jobs is one of the key contributions of private sector to poverty reduction. However, there is limited knowledge about which interventions and in what country conditions are the most likely to catalyze job creation. There is also a lack of clear understanding about which activities are likely to provide most benefit to the poor. To enhance understanding of employment creation based on IFC‘s past results and clients‘ and partners‘ experiences, IFC‘s Development Impact Department (CDI) has been carrying out a study and open source process to elicit lessons that can be applied to strategy making and operations. The study examines the link between IFC‘s activities – both investment and advisory- and job creation and poverty reduction. In particular it reviews how IFC‘s investments/ advisory services help companies create jobs including indirect and induced effects, but net of job destruction across locations, regions, sectors and gender, etc. As part of the study, IFC has conducted this ―meta-evaluation‖ or review of evaluations that were carried out by IFC and other institutions to systematically analyze and report on which activities are most likely to have the largest returns in terms of job creation. More specifically, IFC wishes to know what works best in what circumstances. For example, what types of activities – direct investments, efforts to strengthen supply chains, improving access to finance, infrastructure or the investment climate – are most likely to have the largest effects on job creation? How do these different activities affect different groups of society, such as women, young people or the poor? The objective of the meta-evaluation is to determine the employment effects of private sector interventions and identify the tools/ combination of tools that can be used to estimate their effect on job creation, with a view to help shape the strategies and operations of IFC and those of other private sector oriented development financial institutions (DFIs). Identifying the Relevant Program Evaluations The meta-evaluation began with a review of evaluations commissioned by IFC over the last few years for both Advisory and Investment Services and then was expanded to evaluation work conducted by other relevant organizations. Only completed evaluations were used as the primary source of evidence. More specifically, the meta-evaluation drew information from relevant project level evaluations commissioned by IFC during 2006 - 2011. At the time of the review, 88 internal IFC ‗evaluations‘/ reports were identified for further review of which 11 had a jobs-related component. The following key words were used to identify relevant project evaluations: ―job‖, ―employ‖, ―employee‖, ―employment‖, ―wages‖, ―productivity‖, ―work‖, ―workforce‖, and ―labor.‖ To maximize learning benefits, the scope was extended to evaluations commissioned outside of IFC to those conducted by the World Bank, other multi-lateral institutions, NGOs and research institutes that focused on interventions/sectors similar to those at IFC. The following databases were searched with keywords such as ―evaluation‖ or ―impact evaluation‖, where relevant, combined with ―employment‖, ―employee‖, ―workers‖, ―jobs‖, ―job creation‖, ―jobs provided‖, ―jobs supported‖: 3ie, Abdul Latif Jameel Poverty Action Lab (JPAL), Asian Development Bank (ADB), European Bank for Reconstruction and Development (EBRD), Google Scholar, Inter-American Development Bank (IADB) OVE, International Labor Organization (ILO), JSTOR, Millennium 5 Challenge Corporation, National Bureau of Economic Research (NBER), Network of Networks on Impact Evaluation (NONIE), Science Direct, and the World Bank Development Impact Evaluation Initiative (DIME)1. Initially 59 ongoing and completed evaluations with a job component had been identified outside of IFC. Evaluations that were not yet completed, were public sector interventions, did not actually measure or estimate job creation, or for which copies could not be obtained were excluded. After these exclusions the number of evaluations external to IFC was narrowed down to 28. The evaluations in the final sample to be reviewed were then categorized according to the following four sectors of intervention: Skills Training, Infrastructure, Investment Climate, and Financial Services. Sector Number of Evaluations Skills Training 11 Infrastructure 8 Investment Climate 8 Financial Services 12 TOTAL 39 Central Questions Addressed This review summarizes key findings and identifies major lessons learned from evaluations conducted on private sector interventions and their effect on job creation. The review draws on 3 meta-evaluations, 1 meta-study, 1 meta-analysis, 1 IFC facility review and various types of evaluations that employed different methodologies ranging from focus groups discussions, before and after studies, quasi-experimental to randomized control trials. The review‘s emphasis is on finding out the effects of private sector interventions on job creation and which are the tools/ combinations of tools that can be used to estimate these effects. The main questions that the meta-evaluation addresses are the following: a) What are the lessons/ evidence that link interventions by IFC (or other similar interventions) to job creation in different circumstances? b) How many jobs have been created by private sector/ IFC-type interventions focused on the following sub-categories? 1. Direct effects – how many jobs are ―provided‖ and ―created‖ by IFC clients? 2. What are the indirect effects up and down the supply chain? 3. What are the induced/ secondary effects (through higher income of people) on employment generation? 4. What are the employment effects from the provision of financial services (e.g. better access to finance for SMEs)? 5. What are the employment effects from improved access to infrastructure (e.g. not just direct jobs created, but those in other sectors affected by better access to roads, more reliable power, etc.)? 1 The search was conducted in November/December 2011. Sources and links to the different databases are provided in Appendix 3. 6 6. How do the legal and regulatory changes IFC supports through investment climate programs affect job creation? 7. What are the effects of vocational training and improved skills on jobs? c) What methodologies have been used to answer these questions and what are the strengths and weaknesses of these methodologies? Challenges and limitations of the evaluation universe, data and methodology The findings in this report are based on a synthesis of results from individual evaluations, not all of which had the same objectives or methodologies. For this reason, readers should be aware of certain limitations when interpreting the results presented in the review. The primary limitation is the quantity and quality of the evaluation reports. First, the studies summarized in this report cover only a portion of possible private sector initiatives/ interventions and do not encompass all sectors/ products across institutions. Second, the intensity of the services provided varies across interventions and the objectives of the interventions were not always focused on finding out more about the job creation effects from such activities.2 Thirdly, the methodological approaches ranged from quantitative analysis such as randomized control trials to focus group discussions and qualitative anecdotes from interviews. The evaluations covered Latin America, Asia, Africa, and Eastern Europe, but no evaluations were identified in the Middle East. Moreover, Box 1 below describes more generally some of the challenges in evaluating the impact of job creation effects. Box 1. The challenges of evaluating job creation effects Different job estimation methods. Depending on the research questions and the objectives of interventions, studies focused on different aspects of employment effects such as ―jobs created‖, ―jobs provided‖, ―jobs supported‖, and ―new jobs‖ as opposed to existing jobs, direct, indi rect, induced and net job creation. Moreover, it is often not apparent which effects are being estimated (gross or net effect). Difficulty in what employment to count. There is little agreement in the literature on what employment to count. Some researchers only include paid work outside the household (as a proxy for formal employment) while others use a more inclusive definition that incorporates all types of employment (self, family —paid or unpaid—and paid work outside the household). Difficulty in constructing counterfactuals. Determining the impact of an intervention is often easiest when an evaluator can detect differences based on comparisons between pre- and post-intervention between treatment and control groups. All these require a control or comparable group and may require baseline data that would help illustrate that the treatment and control groups looked comparable before the intervention.3 Capacity, timing, resources, and other considerations often prevent the use of a control group and collection of such data. This issue was identified in many of the evaluations reviewed, particularly those conducted by IFC. Attribution to the intervention. There are often multiple interventions and variables in place that make it difficult to isolate and quantify the effects and link them to specific interventions. Changes could result from one intervention, a combination of interventions, or other development activities by IFC or other donors. 2 For example, the China Secured Transaction evaluation‘s main objective was to increase access to finance for firms, especially SMEs, by developing an appropriate legal and regulatory framework that allows and facilitates the use of movable assets as collateral for commercial loans. Development of the law, development of the security interest registry, increasing public awareness and capacity building were the main outcomes to be evaluated. Job creation was not an objective of the project. 3 In the case of randomized control trials, if the assignment to treatment is truly random and the sample size is large enough, baseline data is not necessary. Randomization ensures that (when samples are large enough, on average) there are no significant differences between treatment and control groups prior to the intervention. 7 Employment effects by private sector interventions Job creation is currently on the minds of just about every policy and decision maker. While the poor consider a job – self-employed or as a worker – as the most important pathway out of poverty (Voices of the poor), unemployment currently stands at over 200 million people (ILO Global Employment Trends 2012). To halve unemployment over the next 15 years, it would be necessary to create over 400 million jobs in Asia and increase employment by over 180 million (70%) in Sub-Saharan Africa (WDR 2013 on Jobs), mainly due to young people entering the labor force. In general, youth unemployment tends to be much higher (on average over 2.5 times the adult unemployment rate) (ILO Global Employment Trends 2012). In this meta-evaluation, we focused on four main areas in which IFC and similar institutions can affect job creation: (i) by promoting access to finance, (ii) access to improved infrastructure, (iii) improvement of investment climate, and (iv) through skill and managerial training. According to the literature, all four are perceived as the main obstacles for firm growth, productivity and thus job creation. The last one is connected to job creation in particular in the case of managerial training. However, it is still a nascent field of research with very little formal evidence. What follows are the main findings from the review of the 39 evaluations by each of these four areas. Access to finance (A2F) Access to finance is one of the key obstacles for private business growth in developing countries and micro, small and medium-sized enterprises (MSMEs) suffer particularly from these constraints. As a result, IFC and other multilateral development organizations have developed and have been implementing programs that address these financial obstacles. To address some of the financial obstacles, various interventions are aimed at improving access to credit, such as investments in or advice to financial institutions that lend to entrepreneurs. There are also advisory service programs to help develop or improve financial infrastructure products such as credit bureaus or collateral registries. Research also shows that developing sustainable financial institutions and improving access to finance can help firms grow in terms of sales, revenues, and operations. The effect tends to be greatest for smaller firms. The findings also indicate that increasing the presence of financial institutions can help create both formal and informal firms. Twelve evaluations were identified and reviewed in this report (five internal IFC interventions and seven external evaluations). Geographically, the evaluations covered the following countries: Bosnia and Herzegovina, Bangladesh, China, Ghana, India, Maldives, Mexico, Mongolia, Morocco, Sri Lanka and Vietnam. The interventions varied from (i) providing savings and loans to micro and SMEs and measuring their effect on socio-economic indicators such as changes in income and changes in enterprise/ business size in terms of number of employees, (ii) developing an appropriate legal and regulatory framework which would allow and facilitate the use of movable assets as collateral for commercial loans with an indirect effect on employment growth, (iii) providing advisory support to partner financial institutions and regulatory authorities to improve outreach to customers, quality of loan portfolio and increase the size of SME/ micro-finance portfolio with an indirect effect on job creation, (iv) providing microfinancing for the first time to underserved populations and (v) providing credit to the poor through rural financial institutions and assessing their effects 8 on household enterprise and employment creation, among other welfare effects such as income and poverty reduction. Despite difficulties in attributing effects on employment directly to the program intervention, all twelve evaluations found positive and/or no effects on job creation. The evaluations also applied different methods of data collection and analysis ranging from using monitoring data, anecdotal evidence, focus group discussions, poverty scorecards to quasi-experimental and experimental designs. Tables 1a and 1b provide a summary of the findings from the twelve evaluations divided into two groups: (i) evaluations without a counterfactual/ control group and (ii) evaluations with a counterfactual utilizing experimental (randomized control trials) and quasi- experimental designs. Table 1a. Evaluations without a counterfactual/ control group Paper Intervention Association with Employment ADB Rural Credit Assistance 39 rural credit projects and 21 Anecdotal evidence that in China, Sri (2001) technical assistance grants in seven Lanka, and Thailand: developing countries. The major - Fujian Soil Conservation and Rural activities supported by the projects Development project: a kelp typically comprised providing access processing plant provided to credit. employment for about 600 female workers who would otherwise be unemployed. - In Thailand about 20% of those interviewed reported changing their main occupation from agriculture. - In other countries, the projects tend to have low impacts in employment generation, which is attributed to implementation difficulties arising from changes in industrial and financial policies. IFC Ghana Microfinance (2011) ProCredit, Ghana is a Savings and Using ProCredit‘s management Loans company that provides information system (MIS) to savings and individual loans to micro construct a baseline, they estimated and SMEs that on average, there was an increase of about 42% permanent hired employees from the beginning of loan one to the last loan. IFC Techcombank SME Banking Techcombank, a joint-stock bank in Borrowers generally perceived that (2009) Vietnam, offered Small Business the loan was important for their Financing (Mi-M5) and Fast Loan business expansion or improvement. (M6) to MSMEs and Business Households (BHHs). IFC Review of SEDF Phase II A2F: Provide AS support to financial The Hikkaduwa Project helped (2011) institutions and regulatory authority. create more than 100 jobs and SBA: AS to increase SME positively affected more than 350 competiveness and linkages. SMEs. Southern Sri Lanka Project – Business Edge Number of jobs supported increased from baseline number of 6000 to 6492 indicating an increase of 8.2% and a shortfall of 1.6% from the target. Kellogg (2009) The study assessed the poverty Using a poverty scorecard, the study profile of workers in BRAC loan found an increase in the number of recipient SMEs in Bangladesh, and employees after an SME received a correlated it to firm characteristics. loan and this effect was more 9 The study also assessed their effect prominent in the trade, on job creation. manufacturing and services sectors (increasing the number of workers from 5-62% compared to last year‘s employment). The increase was more pronounced in rural trading and rural manufacturing. Table 1b. Evaluations with a counterfactual/ control group Paper Intervention Employment Effects IFC Secured Transaction in China The project aimed to increase Job creation effects at the individual Evaluation (2011) access to finance for firms, level of SMEs were inconclusive due especially SMEs, by developing an to the small sample size utilized (38 appropriate legal and regulatory SME beneficiaries vs. the control‘s framework that allows and facilitates group‘s 50 SMEs) in the evaluation. the use of movable assets as collateral for commercial loans. This would be accompanied with the development of a modern secured transactions system in China. Kondo, Orbeta, Dingcong, Rural Microenterprise Finance Participating households have 20% Infantado (2008) Project (RMFP) supported the Govt more enterprises than non- of Philippines to strengthen rural participating ones, and households financial institutions by assisting with program clients have 17 percent organizations that employed the more employed persons than non- Grameen Bank Approach in program clients. providing credit to the poor. Bruhn and Love (2009) New loans to low and middle-income -New bank opening led to an groups in Mexico by Bank Azteca. increase in the proportion of informal businesses by 7.6 percent, but no change in formal. -Total employment, including informal business owners and wage earners, rose by 1.4 percent for the complete sample. - The new bank opening led to higher income levels for both men and women by about 7 percent. Augsburg et al (2012) EKI, a Bosnian microfinance -While overall unemployment of institutions evaluated both households did not change, self- profitability of expanding its employment did increase by 6%. operations to the poorer sector of the -Clients who received the loan were population that it did not serve before also six percent more likely to own a and if that expansion in fact had a business. beneficial impact on new client -Those who had less than primary group. education were more likely to start a business in the agricultural sector compared to similar subcategory of the control group. While those with more than primary education were more likely to start an enterprise in the service sector. Attanasio et al. (2011) Across 40 villages a Mongolian MFI -In case of group loans, those in the XacBank provided group or treatment group have increased their individual loans to poor women who consumption of food and were 10% were randomly selected out of 1148 more likely to own an enterprise by participants. In case of group loans the time of the follow-up survey (1.5 women were encouraged to form years later). groups of 7-15. -In case of individual loans, no 10 statistically significant differences between treatment and controls in the likelihood to own a business. -For individual loan villages, there is an increase in enterprises owned jointly with a spouse. -No statistically significant effect on labor supply, wages or income for both individual and group loans. Crepon et al (2011) Al Amana, the largest microcredit -No impact of microcredit, on the institution in Morocco, opened 60 creation, the profit, or the expansion new branches in non-densely of non-agricultural activities. populated areas. - Households in treatment villages hire significantly more employees from outside the households (19 days per year on average, or 52% difference.) This though comes from agricultural activity. Banerjee et.al. (2010) The effect of introducing microcredit The intervention increased total MFI in a new market. Half of 104 slums borrowing. Fifteen to 18 months in Hyderabad, India were randomly after lending began in treated areas, selected for opening of an MFI the number of businesses increased branch while the remaining were not. by one third. Detailed description of the twelve evaluations: Different methodologies have been utilized across the twelve evaluations. For example, the Ghana Microfinance (2011) project used the client‘s (ProCredit) internal data from its management information system (MIS) to construct a baseline and estimate the number of employees that were hired as a result of accessing the loan from the company. By using this data, they calculated that nearly 15 percent of ProCredit‘s clients had added employees to their workforce between receiving separate loans and about 3 percent decreased the number of employees during the same period. On average, the number of permanent hired employees from the beginning of the first loan to the last loan increased by 42 percent. But as stated in the study, these figures should be taken with caution as they cannot attribute job creation directly to the use of the loan from ProCredit – the IFC client. Poverty scorecards have also been used to determine the employment effects from providing loans to SMEs. A study of BRAC Bank loan recipients (SMEs) used standard regression techniques to derive a firm-level scorecard to estimate the poverty rate of these firms‘ employees and correlated it to specific firm characteristics such as the firms‘ economic sector, its rural or urban location, the portion of unskilled workers, whether the firm employed women, and the total number of workers. The study found that there was an increase in the number of employees after an SME received a loan—and this effect was more prominent in the trade, manufacturing and service sectors (increasing the number of workers between 5-62% compared to prior year‘s employment). The increase was more pronounced in rural trading and rural manufacturing. Moreover, manufacturing firms employed roughly three times as many employees per firm as compared to the trading or agriculture sectors. The trading sector showed the lowest incidence of employing women. The study also found that most female employees worked in the manufacturing sector. Using a different approach, the Vietnam Techcombank SME Banking (2009) review relied on interviews with bank clients to conclude that borrowers in general perceived that the loan provided was important for their business expansion and improvement “facilitating them to create job opportunities for their relatives and local labor”. Anecdotal evidence was also used 11 in the ADB Rural Credit Assistance (2001) evaluation to point to cases of employment generation by projects in China, Sri Lanka and Thailand. For example, the Fujian Conservation and Rural Development project, a kelp processing plant, stated that the project provided employment for about 600 female workers who would otherwise be unemployed. Other countries where the project was being implemented, on the other hand, claimed low impacts in employment generation due to implementation difficulties arising from changes in industrial and financial policies. Yet, it must be noted that very little evidence was provided to back up these figures and findings are perception-based only. IFC also conducted a review of the Small Enterprise Development Facility (SEDF) Phase II program4 located in Sri Lanka and Maldives to assess the overall performance of the projects, assess achievements, validate monitoring data and identify lessons learned. Results from the A2F component were focused on increasing outreach to customers, increasing the SME/ Micro- finance portfolio, improving the capacity of staff and the quality of loan portfolio, and setting up sustainable financial institutions. It was found that the A2F projects were successful in increasing access to finance but no job creation effects were reported. On the other hand, when an A2F sub-component such as training in financial literacy was included as part of the overall intervention, results showed positive economic effects for SMEs through employment generation and increased revenue. For example, it was reported (and validated) that the Tourism Development Project in Hikkaduwa, Sri Lanka which linked SME suppliers of goods and services with hotels/ resorts and helped develop skills and capacity of SMEs in areas such as improved financial practices and better guest relationships resulted in more than 100 jobs created and positively affected more than 350 SMEs. Although 60-80% of program beneficiaries that were interviewed in an SME survey to validate the data attributed more than 50% of their business success to the IFC program, these results should be taken with caution as this was a one-time survey without a counterfactual or control group to robustly support such findings. Other studies such as the China Secured Transaction (2011) evaluation did attempt to construct a counterfactual and used quantitative techniques to estimate the effect of the intervention on job creation. Results showed that as a group, the surveyed SMEs that received accounts receivables (A/R) financing increased their workforce by 7 percent (median) and 36 percent on average in 2011. Yet job creation effects at the individual level of SMEs were inconclusive due to the small sample size utilized (38 SME beneficiaries vs. the control‘s group‘s 50 SMEs) in the evaluation. In other words, the evaluation was not able to provide statistically sound results regarding the overall project‘s impact on job creation from the firm-level data.5 On the other hand, the results were positive when SMEs were surveyed regarding the perceived impact of the intervention. 88 percent of surveyed SMEs said that business growth was a benefit resulting from obtaining A/R financing and 21% of those surveyed thought that growth in employees was a benefit. In the Philippines‘ rural microfinance project, Kondo, Orbeta, Dingcong & Infantado (2008) did apply a quasi-experimental approach using propensity-score-matching with difference-in- differences to find that the take-up of the program loans significantly affected the number of 4 The IFC facility review consisted of desk reviews of activity plans (7 projects under A2F and 5 projects under SBA), interviews with project representatives and IFC staff, an SME survey, focus group discussion with program beneficiaries and project case studies. 5 The evaluators state that they were not able to construct a valid control group (the samples were different sizes and the individual SMEs could be very different from each other). Pairwise matching was not possible and the matching was done only at the aggregate level. Other government policies targeting SMEs during the project lifetime may have also played a role in influencing project effects on job creation. 12 enterprises households have as well as the number of employees in these enterprises. That is, the impact of the program on both the number of enterprises and the number of employed persons in these enterprises was significantly positive. Households that participated in the program have 20% more enterprises than those that did not participate in the program, and households with program clients have 17 percent more employed persons than non-program clients. Bruhn and Love (2009) also apply a quasi-experimental approach using difference-in- differences to comparing outcomes such as informal business ownership, formal business ownership, employment, number of people above minimum wage, and log monthly income in municipalities with and without an Azteca bank branch which would cater to low and middle- income groups in Mexico.6 Opening of the bank tended to have more positive effects on the creation of informal businesses, most likely due to its low documentation requirements for loans which made it different from the majority of already existing lenders. The authors found that new bank opening led to an increase in the proportion of informal businesses by 7.6 percent, but no change in formal. Total employment, including informal business owners and wage earners, rose by 1.4 percent for the complete sample. The new bank opening led to an almost 7 percent increase in the income levels for both men and women. Furthermore, there was a higher proportion of women working as wage-earners in municipalities with Azteca branches after its opening. Augsburg et al (2011), on the other hand, used randomized control trial design to assess the impact of expanding loan operations of Bosnian microfinance institution to the poorer segment of the population. The authors estimated the impact of the loan on business creation and expansion of existing small enterprises and found that while overall unemployment of households did not change, self-employment did increase by 6% compared to the control group. Clients who received a loan were also about 6% more likely to own a business a year later.7 In the experiment, 1206 individuals who would normally be rejected a loan were told that they would have a 50% chance of receiving one if they agreed to be interviewed and then re- interviewed in a year. After the whole group was initially interviewed, half of them were randomly selected to receive a loan. There was heterogeneity of impacts by education level with higher educated benefiting more from this intervention. This could be explained by the fact that those with higher levels of education were more likely to set up a business in the service sector, while those with less education were more likely to set up a business in the agricultural sector. These results are pretty strong also given that the evaluation was conducted at the time when Bosnia and Herzegovina was going through a financial crisis. Banerjee et. al. (2010) also assessed the effect of introducing a microcredit in a new market in India through randomization. The authors found that the intervention increased the total micro-finance institution borrowing. Moreover, 15- 18 months after the lending began in treated areas, the number of businesses increased by one third. In Mongolia, Attanasio et al (2011) also conduct a randomized control trial combined with difference-in-differences and instrumental variable to assess impacts of group loans versus individual loans across 40 villages comprising of approximately 1150 poor women. The authors assessed the following outcome indicators: enterprise creation and growth, changes in 6 In October 2002 Banco Azteca opened branches in all of the existing stores of its parent company – a large retailer of consumer goods, Grupo Elektra establishing more than 800 branches at once. 7 Attrition was surprisingly low as 995 completed a follow-up interview a year later. However, it was systematic as those who did not receive a loan were 10% less likely to complete a follow up interview. 13 enterprise profits and changes in labor supply of affected women. While there were positive effects for group loans, effects for individual loans were only statistically significant for joint ownership. In the case of group loans, those in the treatment group increased their consumption of food and were 10% more likely to own an enterprise by the time of the follow-up survey (1.5 years later). For group lending, profits were also found to be more likely to increase, while no effect was detected for individual lending. The indicated increase in business ownership was largely driven by less educated women (29% more likely to own a business). Additionally, there was evidence that results were likely to be driven by women who were exposed to repeat loans. In the case of individual loans, no statistically significant differences were found between treatment and controls in terms of consumption and likelihood to own a business. However, for individual loan villages, there was an increase in enterprises owned jointly with a spouse. Another randomized evaluation assesses the impact of Al Amana, the largest microcredit institution in Morocco, opening 60 new branches in non-densely populated areas. The authors used randomized control trial with 81 pairs of matched villages.8 And found that the main effect of the increased access to finance from Al Amana was expansion of the scale of already existing self-employment activities of households in the agricultural sector. Households in treatment villages hired significantly more employees from outside the households (19 work days per year on average, or 52% difference) than in the control villages. However, they do not find any impact of microcredit on the creation, the profit or the expansion of non-agricultural activities. In general, quantitative evaluations of microcredit institutions – quasi and experimental – show that improving access to finance does have a positive effect on business expansion and job creation in particular through increasing likelihood of business ownership for poorer segments of the population. In the case of microloans, the effects differ according to the level of education. For example, in Bosnia and Herzegovina the provision of a loan was more beneficial for people with higher education, while in Mongolia the opposite was the case. Additionally, there was evidence in favor of providing group loans versus individual loans where business growth and increase in profit were more likely to occur. Lastly, in case of rural microfinance loans, loans tended to lead to the expansion of the existing businesses in agricultural sector, but had no effect on new business creation in the same sector or on either creation or expansion in the non-agricultural sector. Access to Infrastructure (A2I) Access to infrastructure, particularly power is considered among the top constraints inhibiting firm growth in developing countries. Infrastructure projects tend to have effects far beyond direct jobs created during construction and operation phases. This happens because improvements in infrastructure can lead to growth and creation of firms in other sectors. For example, road projects can contribute to firms having easier access to different markets resulting in a larger number of clients while energy projects provide firms with more reliable access to power allowing for an increased productivity and growth. However, while infrastructure projects are likely to have significant economy-wide job creation effects, their complexity and difficulty in finding appropriate counterfactuals make it challenging to estimate the effects. 8 However, the analysis is done at the household level within these villages resulting in approximately 5,500 observations. 14 In the process of our review, we identified one meta-evaluation and seven evaluations of infrastructure projects that addressed employment effects. Employment indicators tracked included ‗people employed‘, ‗employment generated‘, ‗employment growth‘, ‗wages‘, ‗income‘, ‗probability of hiring a worker‘, ‗worker productivity/efficiency‘, and ‗change in occupational patterns‘. All of the evaluations report positive employment effects from infrastructure projects. One evaluation even analyzed the distributional effects of infrastructure on employment generated among different sectors/occupations. To capture and assess economy-wide employment effects, at least two of the evaluations used new business formation associated with an increase in access to infrastructure as the proxy for jobs created as a result of the project. Tables below present a quick summary of evaluations of various projects, excluding meta- evaluations and meta-analysis. They are classified into two groups: (i) the ones without a counterfactual that are valuable for highlighting relevant trends and factors and (ii) the ones with proper counterfactuals (achieved through randomized control trial and quasi-experimental designs) that allow for credible impact estimates. Table 2a. Evaluations without a counterfactual/ control group Paper Intervention Association with Employment ADB Power Sector Indonesia Multiple with focus on: Surveys indicate that in all selected (2003)  Diversification of generation mix locations there was employment away from oil generation during project  Promotion of rural electrification construction and later in power and private sector participation service delivery. Supply and end-use efficiency improvement ADB Road Sector in Nepal (2000) Road upgrading and periodic Interviews indicate that there was an maintenance of the East-West increase in employment Highway (EWH) as well as two opportunities in general construction access routes into the hills. and related services. Strengthening of institutions in the While net change in jobs was road sector. positive some local craftsmen reported a decline in demand due imports of cheaper manufacturing goods. Suape Port PPP (2011) Installation of Panamax container Increase in number of employees cranes. from 172 in the first year of concession (2002) to 417 in 2011. Using multiplier effects presence of the terminal was associated with the creation of approximately 6,200 additional jobs per year through indirect effects. Table 2b. Evaluations with a counterfactual/ control group Paper Intervention Employment Effects Mu, Walle (2007) Road rehabilitation and new road 2% decline in households relying on development in rural communities in farming as their main source of Vietnam. income 1.7% increase in households relying on the service sector. 15 Lokshin, Yemtsov (2008) Road infrastructure rehabilitation Off-farm employment is 9 projects in Georgia approximately 1.6 percentage points higher for treated villages. Klonner, Nolen (2008) Roll-out of the mobile phone 15 percentage points increase in network coverage in South Africa employment when a locality receives network coverage. Datar, Del Carpio (2009) Irrigation rehabilitation project in Poor farmer‘s probability of hiring rural Peru agricultural workers increases 41 percent more compared to that in the control group. The evaluations identified cover five major infrastructure sectors: power generation (one meta- evaluation and one evaluation), roads (three evaluations), cell phone networks (one evaluation), water (one evaluation) and ports (one evaluation). Geographically, the evaluations covered the following countries: Brazil, Peru, Vietnam, Indonesia, Nepal, Georgia, and South Africa, while the meta-evaluation covered a global range of projects. Methodologically, two out of the seven evaluations are qualitative (ADB (2003), ADB (2006)), with the rest use a quantitative approach, but not always based on a comparison with a counterfactual (Suape Port PP). Results thus should be interpreted with caution. Power sector In the power sector, we were able to identify one qualitative evaluation from the Asian Development Bank (ADB) that utilized focus group discussions and interviews as the means of assessing effects on employment addressing direct and indirect effects of power projects in Indonesia; and one meta-evaluation of 33 studies in the energy sector by the World Bank. The ADB‘s evaluation indicated that new employment opportunities were generated during the project‘s construction phase and later by the power service delivery sector. These effects were not quantified (beyond ―positive‖) and the assessment is perception based. Furthermore, gain in employment that could result from the increased number or growth of existing businesses due to electrification was not assessed. Thus, the effects addressed in the evaluation are just a part of overall positive job creation impact of improved access to power. On the other hand, the meta-evaluation addressed assessment of job creation effects of energy sector projects (in particular alternative energy) focusing on the methodology utilized to derive multipliers for estimation and prediction of economy-wide impacts. The authors point out that most of the studies reviewed lacked a proper counterfactual and neither compared projects costing the same amount on both employment and output metrics, or compared projects producing the same output on both employment and cost metrics. Furthermore, the authors highlight the difference between gross and net (total number of jobs created minus the ones created under a proper counterfactual scenario) impacts using a study of the Solar America Initiative. The gross employment impact of solar production was estimated to be about 50,000 jobs (direct, indirect, and induced) by 2015. However, when the study compared targets of the Initiative to the alternative of providing the same amount of electricity generation through new gas fired plants, the net impact was estimated to be only about 16,000 jobs with direct net employment being negative when compared to the counterfactual. 9 Lokshin, Yemtsov (2003) review 546 rehabilitation projects that are grouped into schools, road infrastructure, water supply, medical facilities and kindergartens, and other. However, we focus only on road rehabilitation projects. 16 Roads sector In the roads sector, we identified three evaluations: (i) an evaluation of ADB‘s road projects in Nepal that used focus groups and interviews; (ii) an evaluation of road rehabilitation projects in Georgia (Lokshin & Yemtsov (2003)) and; (iii) an evaluation of road rehabilitation projects in Nepal by Mu & Walle (2007). The latter two used difference-in-differences approach to quantify projects effects on employment. The ADB‘s evaluation in Nepal indicated positive employment effects generated by the projects and stated that “associated with increase in the number of trading outlets and other developments in the area such as installation of electricity and communication services, employment opportunities in general construction and related services have increased.” However, these positive effects are not quantified and are perception based. The other two evaluations exploiting regional variation constructed proper counterfactuals using propensity score matching and identified an increase in off-farm employment in affected rural areas, which might have indicated creation of non-agricultural jobs associated with the project. Lokshin & Yemtsov (2003) additionally point out that there is an increase in the share of small and medium non-farm enterprises associated with road rehabilitation projects and increase in female employment compared to non-affected areas. Cell phone networks In the case of cell phone networks (or communications infrastructure), we identified one evaluation by Klonner & Nolen (2008) who used instrumental variable technique to evaluate the effects of cell-phone coverage roll-out in South Africa. They tracked changes in occupations, off- farm employment, wages and self-employment as relevant indicators. The authors found that in fact employment increased by 15% when a locality received network coverage. Most of this increase was due to employment of women. There was also evidence in favor of people (in particular young men) switching away from employment in agricultural occupations.. However, there were no effects on wages and on self-employment. It would be useful to understand through which channels employment increased and in which sectors, and what types of firms created jobs in connection with expanded mobile network coverage. Water sector In the case of water sector, only one evaluation by Datar and Del Carpio (2009) was identified. The authors used regression discontinuity and qualitative methods to assess the impact of a large water irrigation infrastructure rehabilitation project in Peru on employment outcomes of poor farmers. They tracked effects on income, poor farmer‘s probability of hiring agricultural workers, and efficiency of labor. The authors found that as a result of the irrigation infrastructure project, poor farmer‘s probability of hiring agricultural workers increased by 41% compared to the control group. Additionally, poor farmers engaged in 35% less manual labor relative to the control group. Moreover, poor farmers increased their total income by more than $220 per year compared to the control group, while rich farmers did not experience such an income gain. Lastly, the authors found evidence that rehabilitation projects improved efficiency of labor through a reduction in time spent in maintaining and operating the irrigation system. Ports sector Lastly, IFC‘s internal evaluation of Suape Port Private-Public-Partnership (PPP) Advisory Mandate does find an increase in employment associated with the presence of the container terminal in Suape Port in Brazil, with direct employment increasing from 172 to 417 from the time of the first concession in 2002. Results from surveys conducted from three firms for which the container terminal in the port was deemed indispensable indicated average employment growth of about 35 percent. Furthermore, by using input-output tables, the indirect employment on the economy is estimated to be an additional 6,205 jobs per year . However, economy wide analysis does not account for any potential confounding factors and in the case of direct 17 employment, it does not employ a proper counterfactual. Thus attribution might be difficult to support in this case. Heterogeneity of effects Notably there was considerable heterogeneity of impacts of infrastructure projects in the roads and cell phone network sectors. They were addressed by four of the evaluations (ADB (2000); Mu & Walle (2007); Lokshin & Yemtsov (2003); Klonner & Nolen (2008)). First of all, while having overall net positive effects, road projects tended to help some sectors such as services grow while depressing others such as local manufacturing through higher competition with imports. For example, evaluation of road sector projects in Nepal by ADB states that ― some local craftsmen, such as tailors and shoemakers, reported a decline in demand for traditional skills, brought about by the import of cheaper manufacturing goods.‖ Second of all, road projects can have stronger or weaker impacts for poor communities depending on the country. In the case of the road rehabilitation projects in Vietnam, Mu & Walle (2007) found that poorer communities tended to experience higher impacts due to lower levels of initial market development, while for road projects in Georgia, Lokshin & Yemtsov (2003) found the opposite with the only exception that female wage employment effect was stronger for poor households. We only had one study on the mobile network roll-out so could not perform a comparison across countries, but within South Africa the employment effects were stronger for poor households and in fact the strongest for women with more than two children, who observed the increase in employment in order of 64 percentage points. Investment Climate It has been shown that the private sector can play a prominent role in the growth process of an economy, but its performance can be significantly affected by the business climate. As noted in the literature, economic and regulatory uncertainty, deficiencies in law enforcement and infrastructure bottlenecks can be main barriers to enterprise development. To reduce these constraints, the Investment Climate Advisory Services program of the World Bank Group provides technical assistance and advice to countries seeking to improve their investment climate. Technical advice is provided in different areas ranging from legislative and regulatory reforms such as business taxation, license simplification and business entry regulation to institution building, investment promotion and reforms in labor market regulations, among other activities. In this review, we were able to identify eight evaluations which attempt to assess how changes in laws and regulations through investment climate reforms could have an effect on job creation. Three of the evaluations are in the Latin America region, one is in Portugal and the remaining four are in Sub-Saharan Africa. Results vary across all regions and different methodologies were employed by each study ranging from randomized evaluation with encouragement design to simple before-and-after approaches without a counterfactual. Tables 3a & 3b provide a summary of the findings from the eight evaluations divided into two groups: (i) evaluations without a counterfactual/ control group and (ii) evaluations with a counterfactual utilizing experimental (randomized control trials) and quasi-experimental designs. 18 Table 3a. Evaluations without a counterfactual/ control group Paper Intervention Association with Employment IFC Africa Investment Climate Investment Climate interventions: The number of jobs associated with Liberia (2011) the reforms supported by the -Business Registration Program was estimated at 16,300- -Investment Promotion 20,400 jobs created during the -PPD and Communications 2008-2010 period. -Doing Business Reforms -Trade Logistics -SEZ IFC Africa Investment Climate Investment Climate interventions: The increase in the number of new Rwanda (2011) businesses also reverberates on -Doing Business Reform employment levels. Based on -Business Licensing Reform average parameters for employment -Improving trade logistics levels in newly established -Investment Promotion & Facilitation enterprises, the acceleration in the -Improving Business Tax enterprise formation process is Administration estimated to have led to about -Simplifying Access to Business 14,900 to 17,700 incremental jobs. Land Adding some 300 jobs attributable to -Facilitating PPD and Reform investment promotion activities, Communication the additional employment associated with Project supported reforms can be estimated on the order of 15,000 to 17,600 jobs. This accounts for between 1.5% and 1.8% of total employment in the country. IFC Africa Investment Climate Investment Climate interventions: Based on average parameters for Sierra Leone (2011) employment levels in newly -Starting a Business established enterprises the number -PPD of jobs in some way associated with -DB Reforms the reforms supported by the Project -Tax can be estimated between 13,500 -Investment Promotion and 16,800 for the entire 2008 – -Tourism Development 2010 period. In the medium term, the employment impact of the Project is expected to further increase as a result of the number of direct jobs created in connection with investments facilitated by the SLIEPA. IFC Africa Investment Climate Investment Climate interventions: Based on average parameters for Burkina Faso (2011) employment levels in newly -Starting a Business established enterprises, the number -Dealing with Construction of jobs somehow associated with reforms supported by the Project can Permits be estimated on the order of 1,700 – -Registering Property 2,000 for the period 2008 – 2010. -Employing Workers This accounts for about 0.3 – 0.4% -Closing a Business of formal employment in the private -Enforcing Contracts sector. No impact on employment -Paying Taxes could be detected in the case of the -Trading Across Borders reform of labor legislation. 19 Table 3b. Evaluations with a counterfactual/ control group Paper Intervention Employment Effects IFC Lima Tracer (2011) IFC provided AS to the Municipality None. of Lima to implement a reform to simplify the licensing procedures in Lima‘s Cercado. The reform was designed to cut licensing time from 160 days to 1.6 days and to lower the number of visits to the municipality (from about 11 time) and reduce the average number of inspections from 5 to one. Branstetter, Lima, Taylor, Venacio In 2005, Portugal implemented a The reform has increased the (2010) one-stop shop type of business number of business start-ups by regulation that offered prospective approximately 17% and created entrepreneurs with significantly approximately 7 new jobs per reductions in administrative fees, and 100,000 county inhabitants. simplified incorporation procedures. Bruhn (2008) Business Registration Reform: Employment in eligible industries establishment of an one-stop shop to increased by 2.8 percent after reform facilitate business entry in Mexico. (in particular, people who were previously unemployed or out of labor force were more likely to work as wage earners after the reform.) Fajnzylber, Maloney & Montes- They explore the effects of a The SIMPLES program suggest that Rojas (2010) business tax reduction and it is possible to increase levels of simplification scheme (SIMPLES) on registration by non trivial amounts formality of micro-firms, and its effect and that this, in turn, leads to much on firm performance indicators. higher revenues, employment and profits among firms which register as a result. Detailed description of the eight evaluations: Bruhn (2008), in her review of the business entry reform in Mexico, estimated that the introduction of one-stop shops to facilitate business entry increased employment by 2.8 percent. In particular people who were previously unemployed or out of labor force were more likely to work as wage earners after the reform. However, the author did not find any evidence of increase in income for those wage earners who opened the business. Other studies have also shown that combining entry reforms with other regulatory reforms such as investment promotion and trade logistics could be associated with job creation. Four evaluations conducted by IFC Africa Investment Climate program in Liberia, Rwanda, Sierra Leone and Burkina Faso estimated between 46,300 – 56,900 jobs created in total in these four countries, although with a caveat that the increases in employment can only be somewhat associated with the reforms supported by IFC projects. The comparative report specifies the following limitations with the studies: (i) they did not account for other donor initiatives/ interventions that could have also contributed to the impacts during the project implementation timeframe; for example, in the case of Rwanda, the decision to reform the business registration system was made prior to the launch of the IFC project and external assistance continued in subsequent years post-IFC intervention, (ii) a proper counterfactual was not possible in any of the cases and only simple before-and-after frameworks were applied which the authors state ―tend to overstate the impact of IFC interventions‖, (iii) different sources of data were utilized depending on availability, and (iii) the timeframe of the evaluations was not sufficient to estimate the final job creation effects from the reforms. 20 Summary of Impacts on Employment Generation 2008 – 2010 (number of jobs) Country Developments in Enterprise Formation Burkina Faso 1,700 – 2,000 Liberia 16,300 – 20,400 Rwanda 14,900 – 17,700 Sierra Leone 13,400 – 16,800 Total 46,300 – 56,900 An evaluation of a similar program in Portugal (Branstetter, Lima, Taylor, Venacio 2010) that used difference-in-differences methodology also looked at another important question from the position of job creation. Namely, ―What kinds of firms enter once the barriers to entry are lowered?‖ The authors found that firms that enter after the reform are about 4.4% less likely to survive their first two years than comparable firms prior to the reform. So despite the fact that the reform increased the number of business start-ups by approximately 17% and created approximately seven new jobs per 100,000 country inhabitants, the duration of these jobs was not clear. Nevertheless, even though some firms tended to exit the market after the reform, this can also be very important as it leads to a constant reallocation of resources/jobs to more efficient enterprises in the economy. Another evaluation commissioned by IFC in Lima, Peru that assesses the impact of license simplification on informal firms found that operating with a municipal license had no statistically significant effect on firms´ performance indicators. Neither final outcome variables (outputs), such as income and profits per worker; nor intermediate outcome variables, such as number of employees, access to credit, and investment in infrastructure and machinery (inputs) were statistically affected by the firms operating with licenses. The authors conclude that results suggest that there may be a correlation between holding a license and performance indicators, in the sense that those holding a license have better performance indicators, but there is no causal link from having a license to performance indicators. ―This interpretation is consistent with a story where growth oriented micro firms will get the license, possibly to avoid legal problems, but not because the license is a key input for growth.‖On the other hand, it must be noted that the authors only looked at the effect of license simplification on informal firms, which is different from the effect on new firm creation (i.e. people who did not previously run a business and started a new business due to the reform). That is, even though little effect was found on the study sample, the reform may still have incentivized people to start a new business contributing to new firm creation. Lastly, we reviewed a study that examined the effect of a business tax reduction and simplification scheme ―SIMPLES‖ in Brazil on the formalization of micro-firms and their effect on firm performance. The SIMPLES program suggested that it is possible to increase levels of registration (formalization) by non trivial amounts and that this, in turn, lead to much higher revenues, employment and profits among firms which register as a result of the program. Yet, it was not clear which part of the program was behind these effects- whether reduced registration costs, reduction of the number of transactions or the overall level of taxation on labor. The results also suggested that the reduction in social security payments for hired labor were central to firms: the lower amounts made firms more willing to register workers, and hence made them less concerned about being detected by government inspectors. Lastly, the results showed that firms born under SIMPLES adopted production technologies and lines of business that were 21 more permanent, capital intensive, and of a larger scale, as measured by the number of employees. Training and skills development Lack of properly trained workers and managers can be a concern for firms that want to grow. In fact, inadequate skill training was cited among the things that have been believed to result in the uprisings in the Middle East and is one of the dominant concerns when it comes to youth employment. According to O‘Higgings (2001), adequate skill training and certification is perceived as one possible remedy in the fight against youth unemployment. Thus education, vocational and skill training affect the supply side of the labor market. At the same time training for managers and entrepreneurs can affect job creation and thus demand side of the labor market as there is evidence that managerial capital is one of the determinants of firm growth and employment. However, this area of research has been rather underdeveloped and the effects of very few programs for managerial training have been properly evaluated (Bruhn et al(2010)). During this review, we identified two meta-evaluations, one meta-analysis, and eight evaluations that address employment effects of training programs. The interventions covered are managerial training (two evaluations), youth training (4 evaluations and 2 meta-studies and one meta-analysis), training for unemployed (one evaluation and two meta-studies), and retraining programs (one evaluation and two meta-studies). The indicators most frequently tracked by these programs are ‗probability of finding employment‘, ‗probability of finding employment in the formal sector (―formality‖)‘, ‗effects on earnings and hours work‘, ‗quality of the job‘, and ‗probability of self-employment‘. Combination of in-class and on the job training is evidenced to be the most effective. However, there is large variation in content and intensity of the training programs so results are not always easily comparable. Even for comparable programs their effectiveness measured by parameters above varies depending on the country and target population. Thus programs have to be country specific. Furthermore, within the same country due to different quasi-experimental designs applied to the same program one might find different results There is also heterogeneity in impacts depending on the gender of trainees. Majority of programs among those reviewed benefited women much more than men. Predictably youth training interventions are more efficient when they target low income youth. Lastly while there is no evidence supporting positive effects of managerial training on job creation via firm expansion in the short-run, there is some evidence of these programs increasing firm profitability that suggest possibility of positive effect on employment in the long run10. Evaluations of training programs that we identified are all quantitative relying mostly on experimental or quasi-experimental design with only one evaluation also including focus group discussion as a part of their methodology (IFC Dialog 2011). Geographically, the evaluations were concentrated in Latin America and the Caribbean and Eastern Europe. Tables 4a & 4b below present a quick summary of evaluations of various projects, excluding meta-evaluations and meta-analysis. They are classified into two groups: (i) the ones without a counterfactual that are valuable for highlighting relevant trends and factors and (ii) the ones with proper counterfactuals (achieved through randomized control trial and quasi-experimental designs) that allow for credible impact estimates. 10 Bloom et al(2011) and Bruhn et al (2012) 22 Table 4a. Evaluations without a counterfactual/ control group Paper Intervention Relationship with Employment IDB entra21 Program Review Entra21 is a regional program The average employment rate (2009) designed to provide 12,000 across 28 projects evaluated was disadvantaged youth ages 16-29 in 54%. The rates varied from a high of LAC with employment training and 82% in southern Brazil to 14% for job placement services youth from Dominican republic. Table 4b. Evaluations with a counterfactual/ control group Paper Intervention Employment Effects Diaz and Jaramillo (2006) Evaluation of a vocational job Employment rates increased by 8.6 training program PROJoven in Peru. percentage points 6 months after The program has two components: training and by 7.3 percentage points (i) learning phase where 12 months after training. training courses are directly Formal employment increased after provided by training centers program participation by 7 to 18 (ii) internship with private firms percentage points. where trainees acquire on-the-job experience. Card et al (2007) Government operated subsidized None training program for low-income youth in urban areas from 2001 to 2005. Program featured several weeks of classroom instruction followed by an internship at private sector firm. Cueto and Mato (2008) Training for unemployed funded by Training increases probability of the European Union in a Spanish being employed by about 8-9%. ―Objective 1‖ region. IFC Dialog (2011) Two-way training workshops aimed None. to strengthen Sri Lanka‘s largest mobile phone operator – Dialog Telecom- network of retail distributors. Benus et al (?) Nine month of training and a small None in Russia. subsistence stipend. The program was implemented in Russia and In Romania the likelihood of being Romania. employed at time of the survey is almost 14 percentage points higher for program participants, controlling for other effects. And likelihood of being employed at least once after getting off the register, the impact of the program is almost a 9 percentage point increase in the indicator. 23 Paper Intervention Employment Effects Attanasio et al (2011) Randomized training program for Employment increased by 6.1 disadvantaged youth in Colombia percentage points. that was introduced between 2001 Paid employment increased by 7.1 and 2005. The program provided percentage points. three months of in-classroom training and three months of on-the-job When controlling for pretreatment training to young people between the characteristics: effect on employment ages of 18 and 25 in the two lowest is 5.4 percentage points on paid socioeconomic strata of the employment 6.8 percentage points. population. Bruhn and Zia (2011) Comprehensive business and None. financial literacy program for young entrepreneurs (loan clients of a partner financial institution) in Bosnia-Herzegovina. With exception of financial literacy training, almost of all of the programs have two components: (i) in class/theoretical training and (ii) on the job training/internship. This combination have been proven to increase success of programs by as much as 20% (Fares and Puerto 2009). It is not always clear if the government or private sector or a combination of both provided the training. However, in contrast with results that come from evaluations of training programs in Western Europe and the United States, there does not seem to be evidence in favor of trainings provided by private sector having better results than those provided by the public sector in Latin America. On another note, an evaluation of a training program in Mexico concludes that once the program was restructured so that school based modality was phased-out and on-the-job training in large firms became a requirement, employment effects became larger. Thus, there is some evidence that having a larger on-the-job training component can be more effective in some contexts. The easiest way to establish a link between training and job creation is to look at the effects of training on self-employment11 or growth and survival of existing companies in case of managerial training. However, only two evaluations in our sample address the effects of training on probability of being self-employed for non-managerial training. Neither of them found any conclusive evidence in favor of training increasing the chance of self-employment. The first study is cited in the meta-evaluation by Ibarraran et al (2008) and was looking at outcomes of the Probecat training programs in Mexico. The study found inconclusive effects of training on probability of being self-employed but positive effects on the wages of self-employed trainees. The second study by Attanasio et al (2011) that evaluated effects of subsidizing vocational training for disadvantaged youth in Colombia also found no evidence in support of higher probability of one being self-employed. This, however, should not be taken negatively as it might indicate that training was just efficient in equipping workers with skills needed to find a formal job and thus there was no need for self-employment. Two evaluations12 that are looking at the effects of training managers/ entrepreneurs were identified: (i) IFC‘s internal evaluation of IFC Dialog, a project that c onsisted of two-day training 11 However, this should be interpreted with care as self-employment is in many cases sub-optimal and often done as last resort because people can‘t find a formal job. 12 In addition to this two evaluation, the Peer Reviewer recently identified two additional evaluations (Bloom et al (2012) and Bruhn et al (2012)) that are not included in this meta-evaluation due to the fact that they were completed or the relevant components to this study were included after December 2011. 24 workshops for entrepreneurs working in the cell-phone retail distribution sector of Sri Lanka and (ii) an evaluation of the effects of comprehensive business and literacy training for young entrepreneurs in Bosnia-Herzegovina (Bruhn and Zia (2011)). The IFC evaluation used before- and-after methodology with control and treatment groups along with a qualitative assessment via focus groups. The indicator of interest was number of people employed. On the other hand, Bruhn and Zia (2011) used a randomized control trial to test if training impacted the chance of business survival. Neither of evaluations found statistical support for these programs having an effect on tracked indicators. However, in the case of financial literacy training, there was evidence in favor of improved business practices and higher investment for surviving businesses as a result of training. This might imply that managerial training does have a positive effect on productivity, which is further supported by two more recent evaluations of managerial training that found positive impacts on profitability13. This is consistent with the findings from the evaluation of a supervisory training program in Cambodia of the garment sector which found that training improved work relations and increased output of workers (ideas42, 2011)14. Lastly, assistance programs for self-employed were included in a meta- evaluation by Betcherman et al(2004). In particular four evaluations from developing and transition countries were reviewed and some of the programs indicated to have positive effect on employment, but for older and better educated workers. Training programs for youth have been considered to be more successful in developing than in developed countries and evaluative evidence seems to be primarily focused in Latin America. General findings from evaluations of training programs in the region point out that those programs have modest to no effects. The indicators tracked most often are impact on ‗chance of employment‘, ‗employment in the formal sector‘, ‗earnings and quality of the job‘. In the Dominican Republic, Card et al (2007) do not find any statistical support for the effect of youth training and their chances of finding employment in the Juventud y Empleo program, with the exception of the East and Santo Domingo regions of the country (yet, monthly labor earning of those trained was found to increase by at least 17% and there was evidence of positive and significant impact on job quality). Furthermore, some other studies have found similar effects. According to the meta-evaluation by Ibarraran et al(2008) there was no statistical support for positive probability of employment as a result of training according to evaluations of programs in Argentina (with evidence in favor of improved quality of jobs) and in Panama (but improvement in quality of jobs was not measured). On the other hand, evaluations of the PROJoven training program in Peru (Diaz and Jaramillo 2006), a youth training program in Colombia (Atanasio et al 2008), and a training program in Mexico (Delajara 2006) find support for positive effect on probability of finding employment. Almost all of the evaluations do find a positive effect of training on the probability of finding a job in the formal sector. Effects on employability and income are stronger for women (Panama, Argentina, & Colombia) and for young people (Mexico). These are consistent with later findings from evaluations by Attanasio, et al (2011) of training programs for disadvantaged youth in Colombia. Additionally, there is evidence that programs are more effective for disadvantaged youth. Effects of training for unemployed are addressed by one evaluation (Cueto and Mate 2008), which used propensity-score-matching to analyze the effects of ―Objective 1‖ training program in Spain, and by one meta-evaluation (Betcherman et al2004) that analyzed results from 14 studies in developing countries and 35 in developed ones. According to Betcherman et al(2004) training for unemployed is found to be beneficial in terms of higher employment rates but not in 13 Bloom et al (2012) and Bruhn et al (2012) 14 This evaluation was commissioned by IFC but was not included in the 33 reviewed as it did not have a job creation component. 25 terms of higher earnings. Additionally, it is stated that it was not as effective in developing countries as it was in developed ones and results were more positive for women than for men. On the other hand, Cueto and Mate (2008) find in Spain that the results were better for men than for women and the training was found to increase employment probabilities by 8-9%. However, the authors caution that selection into training might be more important than the actual components of training provided. Effects of retraining programs15 were analyzed in one evaluation by Benus et. al. that used a quasi-experimental design to address retraining programs in Russia and Romania, and one meta-evaluation (Betcherman et al 2004) that reviewed results from evaluations in developed and developing countries. According to Betcherman et al (2004) retraining programs often have no positive impacts with the exception of a couple of cases in China and Bosnia-Herzegovina that included comprehensive packages of employment services and not just a retraining component. Furthermore, in Bosnia-Herzegovina -which was the most successful program- the authors caution that results should not be generalized as the positive retraining effects may have been specific to the post-conflict status of the country and it was unclear what percentage of success was attributable to that particular component of the intervention. The study by Bensus et al. looked at retraining programs in Russia and Romania. The authors tracked the following indicators: likelihood of being employed (short and long term), the likelihood of obtaining a high salary and the length of unemployment as a result retraining program in Russia and Romania. They conclude that training had no statistically significant impact on any of the indicators tracked in case of Russia but had positive effects in Romania. There was no gender variation for results in Russia. On the other hand, in Romania there was evidence in favor of re- training being particularly beneficial for females while having no impact on males. The program was also found to be more beneficial for middle-aged trainees, those with low levels of education and those coming from rural areas in Romania. 15 Note that in some evaluations it is hard to tell what is the difference between retraining programs and vocational training programs for unemployed as different names are used depending on the country . 26 Analysis of evaluation methodology The evaluations reviewed in the four different areas (A2F, A2I, IC and Training) utilized a wide range of evaluation methodologies. They ranged from randomized control trials, quasi- experimental designs with difference-in-differences, difference-in-differences with propensity- score-matching, panel data with fixed-effects, instrumental variables, before-and-after, focus group discussions, interviews, surveys, document review, counterfactual analysis, firm-level scorecards, and a mixture of the above. Additionally, meta-evaluations and meta-analysis were also included in the sample. Ideally, program evaluations would compare the group of program participants with a group of program non-participants (control group) that had similar attributes as those that were benefiting from the program. These two groups of individuals/ firms would then be tracked over time to see if there are any differences in performance indicators to be evaluated. Having a control group or constructing a ‗counterfactual‘ can help evaluators determine what would have happened in the absence of the program and can best be used to assess attribution to the intervention. Issues of selection bias, that is, the extent to which various subgroups or target population are likely to participate differently in a program, can also be controlled for using these types of methodologies. An evaluation should ideally begin with construction of a baseline to clearly define the starting point of program participants. Attribution to the intervention can best be supported once a comparison between program and non-program participants is carried out over time by qualified evaluators. Unfortunately, it is not always possible to construct a counterfactual and/or identify a proper comparison group. There may be various reasons that go beyond the control of the evaluators that could limit the scope of an evaluation. For example, the evaluator may come across the following constraints: (i) incomplete/ outdated contact information, (ii) refusal to participate in a survey, (iii) firm attrition16, (iv) not being able to identify a similar group to interview as they do not exist, (v) sample size restrictions, (vi) budget restrictions, (vii) low levels of available qualified survey firms, (viii) legal, ethical or practical issues, among many others. As a result, qualitative evaluations can complement some quantitative approaches by examining some of the subjective aspects of the services received (e.g. satisfaction rates, perceived outcomes and/ or impacts from the intervention). Having noted these caveats, in this section, we discuss various types of methodologies that were applied by the different authors in the four areas reviewed and what were the tools/ combination of tools that worked best to estimate job creation effects by the various types of private sector interventions. Access to Finance Evaluations in the financial services sector used a combination of quantitative and qualitative methods to estimate the job creation effects of private sector interventions on micro, small and medium sized firms. Twelve evaluations were reviewed. Seven of them were able to construct a proper counterfactual or comparison group and used experimental or quasi-experimental 16 Survival rates for firms are quiet low, especially when it comes to micro, small and medium firms. Firms may exit the market because they are not able to compete in the current business environment but can also exit because entrepreneurs found better opportunities elsewhere. 27 designs. Two evaluations17 did attempt to construct a counterfactual but due to logistical issues, such as not being able to account for baseline data or not being able to access firms that were previously surveyed during the baseline exercise, alternative methods of data collection were used to determine the effect on job creation. These included focus group discussions and individual interviews of program beneficiaries to learn more about their opinion regarding the services they‘ve received and their perceptions on certain program performance indicators such as ―increased job opportunities‖ due to the program and/ or ―growth in employees‖. One evaluation went further and actually validated monitoring data collected semi-annually during supervision periods by members of the program team after the project had been completed. The independent firm hired to validate such results was able to confirm the number of jobs claimed to have been created by the intervention. Moreover, program participants were also surveyed to inquire further if the changes they had seen were a result of the intervention and not due to other factors. These qualitative techniques can serve as useful tools to shed light on the job effects from private sector interventions. Unlike other sectors such as infrastructure or investment climate, in the case of microfinance interventions, a large number of the evaluations showed that it is much more plausible to conduct quasi or randomized experiments. That is, it can be much easier to measure the impact of a new loan product to help individuals or groups expand their business and create jobs than to assess the economy-wide effects of investment climate reforms. Researchers can identify a microfinance institution and individuals or group of individuals to lend to over a period of time and at the same time identify a comparable group (control) that is not benefiting from the intervention (the ―counterfactual‖). The ability to construct a counterfactual is not always the case when it comes to other sectors. Access to Infrastructure In the infrastructure sector, the evaluations were qualitative, before-and-after, or quasi- experimental. Due to infrastructure projects often having significant economy-wide job creation effects evaluators have to find ways to estimate employment beyond jobs created by the client and client‘s supply chain. Additionally, since often projects are being at least partially financed by the government, evaluators should be particularly attentive in identifying a proper counterfactual to account for the opportunity cost of such government spending. As noted, an evaluation should account for an appropriate counterfactual when using a quantitative approach. More specifically, a World Bank meta-evaluation of the power generating sector outlined some of the metrics that should be taken into account when evaluating infrastructure and in particular power sector projects. The authors specify that studies should compare projects that cost the same amount on both employment and output metrics, and/or compare projects that produce the same output on both employment and cost metrics. Moreover, the authors highlight that proper counterfactuals are particularly important in case of infrastructure projects that tend to be financed partially by governments: the analysis should not only evaluate alternative government spending packages, but also the impacts of the opportunity costs of spending (that is, cutting spending elsewhere, raising taxes or borrowing). Furthermore, for energy investments projects another important aspect to take into consideration is the impact on the household budget for consumers. Some energy efficiency products may save households‘ money, while the projects that increase the cost of energy to 17 Ghana Microfinance (2011) and China Secured Transaction (2011). 28 the user (e.g. mandates for renewable generation) will result in reduced household budget for other goods. This in turn will affect induced employment. While in case of infrastructure it is particularly hard and expensive to implement randomized control trials, the closest alternative becomes quasi-experimental approach, e.g. difference-in- differences with propensity score matching. Since often infrastructure projects impact whole regions versus individuals even this approach is often difficult to implement. However, a few studies (in particular in roads, water and cell-phone sectors) in our sample did successfully employ this methodology. It was applied in case of projects where smaller regions were affected and where due to data availability they could more easily be matched with comparable regions that did not receive the treatment. Evaluating job creation effects of infrastructure projects can also be more challenging because a significant part of job creation results from providing other businesses in the area with more reliable access to infrastructure. Thus, effects go beyond client and supply chain. While measuring these effects can be extremely challenging, a few studies do attempt this through the application of input-output (IO) tables. However, caution should be used when interpreting these multipliers as frequently: (i) sectors are not sufficiently disaggregated; (ii) models used are linear and do not allow for either scale nor substitution effects to be accounted for; and (iii) lastly, for many developing countries, IO tables are either not readily available or outdated thus possibly not accounting for technological change. Although IO tables might not be a perfect tool to measure economy wide effects they do provide a useful number that can be used as a benchmark. Furthermore, they have a clear advantage of being relatively easy to compute due to their linear nature. Lastly, four out of the seven evaluations (two in roads, one in telecommunications infrastructure, and one in water sector) show that the employment effects and their magnitude vary depending on gender of workers, industry (some industries might be hurt while other‘s benefit), and socio- economic status of the affected community. In particular, all four attempt to address the effect of interventions on ―poor‖ households but how ―poor‖ is defined varies across the different interventions. Thus one has to be careful as to which definition is being used when making cross-country and/or cross-sector comparisons. Investment Climate All eight evaluations reviewed in the investment climate area used quantitative techniques to estimate the job creation effects of private sector interventions. These included the following: econometric analysis of panel data from employment/ micro-enterprise surveys (two evaluations), experimental and quasi-experimental approaches using differences-in-differences (three evaluations), and before-and-after techniques without a counterfactual (four evaluations)18. All but one of the evaluations had positive effects on job creation, particularly at the economy-wide level. The one that was not able to show positive effects had no statistical effect on employment generation, among other performance indicators. Different data sources were used across the evaluations to analyze the effects from changes in laws and regulation on employment outcomes. These sources ranged from national establishment census data, company registry data to labor market surveys and World Bank 18 The survey questionnaire of these four evaluations included qualitative aspects/ questions that focused on strategic relevance, delivery of outputs, achievement of outcomes, and efficiency. 29 Enterprise surveys. Various surveys were also conducted over different periods of time to ascertain changes in performance indicators before and after the intervention. Even though positive results were claimed from various investment climate reforms such as business entry/ firm registration and investment promotion, in the case of the four pre-post evaluations without a control group, the authors caution attribution to the interventions as other factors may be contributing to those findings. Attribution was less of an issue for evaluations that focused specifically on business entry and tax simplification reforms which used staggered implementation of the reforms with high-frequency data to carefully identify causal effects. Interestingly, the one evaluation that found no effect on key performance indicators such as income or profits or number of employees utilized one of the most sophisticated quantitative techniques (randomized control trial with encouragement design). The authors implemented a panel/ tracer study that followed a sample of micro-firms over a two and a half year period through the application of four rounds of survey questionnaires. Yet the results were inconclusive and the authors claimed this was mainly due to the instrument used19 and the high attrition rates of firms surveyed over the two year period. Moreover, the timing of the data collection should also be taken into consideration when conducting such type of experiments. Training and skill development All the evaluations reviewed under vocational and skill trainings were quantitative and almost all experimental or quasi-experimental in nature. In particular, experimental/ randomized-control trials were used to assess the effects of interventions in the area of vocational and skill development training. Furthermore, when a quasi-experimental approach was used, the authors were able to perform various tests of confidence to make sure that results were not biased due to self-selection into the program or reverse causality. While results were generally reliable and of good quality, the trainings evaluated usually targeted a specific demographic segment of the population which means that results cannot be generalized to the entire population. As noted, a few studies reviewed were able to use randomized control trials to assess the effects of skill training on jobs creation. This in general should assure the most reliable assessment. However, the authors highlight that due to the nature of the interventions and bureaucratic delays in both program and evaluation implementation the following should be taken into consideration when interpreting results: (i) original assignment into treatment and control has not been respected in the implementation of the programs due to self-selection and reassignment of treatment and control (ii) non-random selection into attrition (trainees chose to drop out of the program non-randomly). For example, in the case of a training program in the Dominican Republic by Card & Ibarraran (2007), over a third of the original control group was reassigned to treatment and there was evidence in favor of non-random selection into dropouts and no-shows. Thus results reflect the differences between realized treatment and control groups and not between the groups as they had been initially assigned. Similarly, in the case of training program in Colombia, Attanasio & Kugler (2011) note that there was a possibility that results for men could be biased due to some self-selection into either treatment or control groups and high levels of attrition. The direction of the bias was, however, unclear. Lastly, an important aspect to evaluate is the quality of jobs after training and their duration. However, this can be in particularly challenging as it might require additional follow up studies. 19 Responses may not always be honest. When asked if the firm wants to be formal, overwhelmingly the response would be positive but when the actual option was verbally given, the response would be otherwise. 30 Nonetheless, two out of the eleven evaluations were able to address the quality of the job and its duration. 31 Conclusion and Recommendations Thirty-nine evaluations were reviewed in the following four areas: (i) access to finance, (ii) access to infrastructure, (iii) investment climate and (iv) skills development and training. The evaluations consisted of private sector interventions carried out by IFC and other organization and their effect on employment generation. The job related components focused on job creation, earnings, quality of jobs, occupation patterns, formal sector employment, gender and youth effects, and possible heterogeneity of effects. The evaluations utilized different methodological approaches that were qualitative and quantitative in nature or a combination of both. Evaluations were also geographically located across the following regions: Latin America, Asia, Africa and Eastern Europe. However, no evaluations with job-related components were identified in the Middle East. Twelve evaluations were reviewed in the Access to Finance sector that focused on the provision of loans and advisory services to micro, small and medium enterprises as well as households. The evaluations were conducted in Bosnia and Herzegovina, Bangladesh, China, Ghana, India, Maldives, Mexico, Mongolia, Morocco, Sri Lanka and Vietnam. Different methods of analysis and data collection ranging from anecdotal evidence to experimental designs were utilized. Overall, it was found that improving access to finance can help firms expand their operations, which can have a positive effect on the quality and number of jobs created. The effects tend to be greatest for smaller firms. Combining access to finance with advisory services also tends to have a more positive effect on employment generation. Moreover, through quantitative approaches, it was found that improving access to finance for micro-enterprises can create jobs both through the establishment of new businesses and through expansion of already existing ones. The latter effect tends to dominate in rural settings. Investments in services sector in urban areas and in agriculture in rural areas tend to create the most jobs. Collective loans are likely to have stronger effects on employment than individual ones. Seven evaluations and one meta-evaluation were reviewed under Access to Infrastructure. The job creation effects were overall positive across the different sectors reviewed which were roads, power, water, ports and telecommunication infrastructure. From a methodological point of view, two-thirds of the evaluations were quantitative, but not all were able to construct a proper counterfactual to evaluate the effect of the intervention on job creation. However, in some cases quasi-experimental approach matching treated regions with possible controls was successfully implemented to account for a proper counterfactual scenario. This is particularly replicable in cases where projects affect smaller areas/groups and data is available to match them with comparable non-affected areas/groups. The predominant effects on employment tend to be indirect. Furthermore, some evaluations not only managed to capture economy-wide effects in addition to direct job creation but also differentiated between employment generated by construction and maintenance phases. In comparison to other infrastructure sectors, road rehabilitation projects while having on net positive effects tended to help some sectors such as services grow while depressing others sectors such as local manufacturing through higher competition with imports. These projects also were found to diversify occupation mix in the area away from agricultural projects. Eight evaluations were reviewed under investment climate. These evaluations were conducted in Brazil, Burkina Faso, Liberia, Mexico, Peru, Portugal, Rwanda and Sierra Leone. Seven showed to have had positive economy-wide effects on job creation while the license simplification reform evaluation showed no effect on any firm performance indicators, including income, credit or jobs. The seven evaluations that showed positive effects focused primarily on 32 business entry/registration reform, investment promotion and tax simplification. Overall, it was found that these types of investment climate reforms tend to lead to the creation of new firms which can have positive employment generation effects. Yet firms that enter after the reform may be less likely to survive their first two years than firms that enter prior to the reforms. So even though new jobs may be created, their duration and quality are not always clear. Additionally, the following caveats also need to be taken into consideration when assessing the economy-wide effects of investment climate reforms that were not able to construct a proper counterfactual: (i) all had issues with attribution and could not account for other factors or interventions that could have influenced the results and (ii) due to the nature of the business (ie. legislative and regulatory reforms) it was difficult or impossible to construct a counterfactual, and (iii) the timeframe needed to estimate the effects on job creation is long and often would need to be extended beyond the project lifetime. Eight evaluations, two meta-evaluations and one meta-analysis were reviewed under skills training. The interventions included youth training, training for unemployed, re-training, and managerial training. Evaluations were mainly carried out in the Latin American and the Caribbean region, Eastern Europe and one evaluation in South Asia. Results were mixed. Additionally due to variation in intensity, content, duration and target population of the training programs results are not always readily comparable. Managerial training was not found to have an impact on either the survival of the business (linked to the stability of the job) or on number of employees. However, it was found to have a positive effect on business practices, profitability and investments by the business. This is consistent with an evaluation that showed that supervisory training programs can have a positive effect on employee and manager work relations and worker productivity. Together these can be linked to positive employment effects in the long term. Vocational training for youth was deemed successful in some but not all countries at increasing the probability of being employed after the training received. Wages and future earnings also were found to be higher after the training. The probability of being formally employed was found to be affected positively across all the evaluations that tracked this outcome indicator and job quality also tended to have improved after the training. Combining in- class training with on-the-job training also tended to have positive effects. Vocational training seemed to also be more beneficial for women and disadvantaged youth across all the evaluations reviewed. In conclusion, we recommend that more rigorous evaluations be carried out across all sectors, and in particularly in the area of access to finance when it comes to internal IFC evaluations. Additionally, it would be helpful to harmonize the definition of what kind of jobs qualify as ―employment‖. This would make results more comparable across different programs. Evaluations that encompass a longer time horizons (in particular in the case of skills and training and investment climate projects) would also strengthen the robustness of the effects of private sector interventions on job creation. In the case of investment climate projects, longer evaluation timeframes may be needed to estimate the final job creation effects from the reforms. For skills and training programs, evaluations after longer periods of time will allow for: (i) a better assessment of the stability and quality of employment after the program has ended, and (ii) the effects to take place (in case of managerial and entrepreneurial training.) Regarding the first point, while quality and duration of employment after training may be important metrics, almost no evaluations addressed them. This could be due to relatively short follow-up periods after the training took place. Secondly, there is some evidence that managerial training programs do increase profitability and productivity of the affected firms. These can translate into positive employment effects in the long-run. Thus evaluations with only one follow-up in relatively short periods of time may not be capturing the entire employment effects. 33 Appendix 1: Methodology type by subject area20 Quantitative Qualitative Mixed Access to finance 7 3 2 Access to Infrastructure 5 2 0 Investment Climate 4 0 4 Training 8 0 0 Access to Finance: Quantitative: Rural Microenterprise finance project in the Philippines (2008), Evaluating poverty outreach of small lending by BRAC Bank, Bangladesh (2009), Microfinance at the margin: Experimental Evidence from Bosnia and Herzegovina (2012), Group Lending or Individual Lending? Evidence from a randomized field experiment in Mongolia (2011), The Economic Impact of Banking the Unbanked: Evidence from Mexico (2009, The miracle of microfinance? Evidence from a randomized evaluation n India (2010), Impact of microcredit in rural areas of Morocco (2011) Qualitative: ADB Rural Credit Assistance (2001), IFC Techcombank SME Banking Vietnam (2009) Mixed: IFC China Secured Transaction (2011), IFC Facility Review of SEDF Phase II in Sri Lanka and Maldives (2011), IFC Ghana Microfinance (2011) Infrastructure: Quantitative Evaluations: Suape Port PPP (2011), Road rehabilitation in Vietnam (2007), Infrastructure rehabilitation project in Georgia (2003) , Roll-out of mobile phone network coverage in South Africa (2008), Irrigation rehabilitation project in Peru (2009) Qualitative Evaluations: ADB Power sector projects in Indonesia (2003), ADB Road sector in Nepal (2000). Investment Climate: Quantitative: IFC Africa IC Liberia (2011), IFC Africa IC Rwanda (2011), IFC Africa IC Sierra Leone (2011), IFC Africa IC Burkina Faso (2011) , IFC Lima Tracer (2011), Business entry regulation reform in Portugal (2010), Business registration reform in Mexico (2008), Evidence from the Brazilian SIMPLES program (2011) No Qualitative Evaluations or ones with mixed methodology were identified. Skill Training and Development: Quantitative Evaluations: Youth labor training program in Peru (2006), IDB entra21 program (2009), Youth training in the Dominican Republic (2007), Vocational training for disadvantaged youth in Colombia (2011) , Training for Unemployed in Spain (2008), Re-training program in Russia and Romania IFC Dialog (2011) , Business and financial literacy training for young entrepreneurs in Bosnia-Herzegovina (2011). No Qualitative Evaluations or ones with mixed methodology were identified. 20 Note that meta-evaluations and meta-analysis are excluded from this count. 34 Appendix 2: Summary of Evaluation Results by Employment Indicator This section presents tables for each area reviewed with a breakdown of individual21 evaluations by methodology and effects for selected indicators. In case the evaluation did not track the specified indicator, it is not included in the table. Hence, the tables do not cover all the evaluations reviewed for each subject area/ sector. Table 1. Access to Finance Indicator: Employment creation/number of employees/employment growth22 Methodology/Effects Positive No Effect Mixed Quantitative 6 0 0 Qualitative 2 0 0 23 Mixed 3 1 0 Table 2. Access to Infrastructure Indicator: Employment opportunities, number of people employed, probability of wage employment, number of workers hired24 Methodology/Effects Positive No Effect Mixed Quantitative 4 0 0 Qualitative 2 0 0 Mixed 0 0 0 Table 3. Investment Climate Indicator: Number of jobs created/employment Methodology/Effects Positive No Effect Mixed 25 Quantitative 6 1 1 Qualitative 0 0 0 Mixed 0 0 0 Table 4. Vocational Training for Youth Indicator: Probability of being employed (when differentiated, the indicator is tracking the probability of being employed in the formal sector.) Methodology/Effects Positive No Effect Mixed 26 Quantitative 3 0 1 Qualitative 0 0 0 21 Thus meta-evaluations and meta-analysis are not included. 22 Evaluations of Rural Microenterprise project in the Philippines (2008) and of microfinance loans in Bosnia and Herzegovina are not included as they do not track this indicator. 23 Attanasio, et al (2011) find negative wage employment effect for group loans in Mongolia (likelu due to more women switching to self-employment), and positive wage employment effects for individual loans (likely due to women in this category being employed in joint ventures with their partners.) 24 The evaluation of effects of road rehabilitation projects in Vietnam is not included. 25 Evaluation of the IC reforms in Burkina Faso showed no effect of labor legislation on employment, but all the other reforms did have positive effects. 26 Vocational training in Dominican Republic was not found to have a statistically significant effect on probability of being employed with exception of East and Santo Domingo Regions. 35 Mixed 0 0 0 Table 5. Training for Managers/Entrepreneurs Indicator: Chances of business survival and change in employment Methodology/Effects Positive No Effect Mixed Quantitative 0 2 0 Qualitative 0 0 0 Mixed 0 0 0 Table 6. Training for Unemployed and Retraining Indicator: Probability of being employed Methodology/Effects Positive No Effect Mixed 27 Quantitative 1 0 1 Qualitative 0 0 0 Mixed 0 0 0 27 Effects of retraining program were statistically significant in Romania but not in Russia. 36 APPENDIX 3: Results from IFC monitoring system data Results data from IFC’s Development Outcome Tracking System (DOTS) There has been some analysis carried out using monitoring data obtained from IFC‘s Investment client companies. The data shows that direct jobs created by IFC‘s client companies – net of job losses – tend to be relatively small. The 375 companies that reported employment data from 2008-2010 added about 100,000 jobs, mostly in Asia and LAC. This 10% increase in jobs is still an achievement, considering that globally there were significant job losses. Per million dollar of project costs, some sectors (e.g. wholesale and retail trade, services, health care) added many more direct jobs than others (e.g. heavy industry, or power generation). However, the data should be viewed with caution since it does not take into account: (a) Job losses in competitors (For example, in retail, DOTS data shows job creation of over 60 jobs per $ million; a study suggest that for every 100 jobs created through modern retail, 50 were lost elsewhere, implying that there would still be job growth, but only about half as much as our figures indicate); (b) Jobs in suppliers or distributors; or (c) Secondary effects, e.g. more reliable power resulting in enterprises being able to produce more and more efficiently. On the other hand, in mining, direct job creation per $ million is very small, but as discussed in the previous section, the indirect effects for these industries are significantly higher than for retail and services. Thus economy wide effects of those activities might be higher but we do not yet have information to ascertain this. 37 APPENDIX 4: Sources/ Databases 1. 3ie 2. Abdul Latif Jameel Povery Action Lab (JPAL) 3. Asian Development Bank (ADB) 4. European Bank for Reconstruction and Development (EBRD) 5. Google Scholar 6. Inter American Development Bank (IADB) 7. International Finance Corporation (IFC) 8. International Labor Organization (ILO) 9. JSTOR 10. Millennium Challenge Corporation 11. National Bureau of Economic Research (NBER) 12. Network of Networks on Impact Evaluation (NONIE) 13. Science Direct 14. World Bank Development Impact Evaluation Initiative 38 REFERENCES Asian Development Bank. ―Asian Development Bank Assistance to the Power Sector in Indonesia.‖ Evaluation Reports. Impact Evaluation Studies. July 2003. Available at: http://www.adb.org/documents/adb-assistance-power-sector-indonesia Asian Development Bank. ―ADB's Rural Credit Assistance in Bangladesh, Peoples Republic of China, Indonesia, Nepal, Philippines, Sri Lanka and Thailand‖ Evaluation Reports. January 2001. Available at: http://www.adb.org/documents/adb-rural-credit-assistance-bangladesh-prc- indonesia-nepal-philippines-sri%20lanka-thailand Asian Development Bank. ―ADB Assistance to the Roads Sector in Nepal.‖ Evaluation Reports. Impact Evaluation Studies. December 2000. Available at: http://www.adb.org/documents/adb- assistance-roads-sector-nepal Attanasio, Orazio, Adriana Kugler, and Costas Meghir. "Subsidizing Vocational Training for Disadvantaged Youth in Colombia: Evidence from a Randomized Trial." American Economic Journal: Applied Economics, 3(3): 188–220. 2011. Available at: http://www.aeaweb.org/articles.php?doi=10.1257/app.3.3.188 Attanasio, Orazio, Britta Augsburg, Ralph De Haas, Emla Fitzsimmons, and Heike Harmgart. ―Group Lending or Individual Lending? Evidence from a randomized field experiment in Mongolia.‖ Working Paper No. 136. European Bank for Reconstruction and Development. December 2011. Available at: http://www.ebrd.com/downloads/research/economics/workingpapers/wp0136.pdf Augsburg, Britta, Ralph De Haas, Heike Harmgart, and Costas Meghir. ―Microfinance at the Margin: Experimental Evidence from Bosnia and Herzegovina.‖ Version from March 2012. Available at: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2021005 Benus, Jacob; Raluca Brinza, Vasilica Cuica, Irina Denisova and Marina Kartseva. ―Re-training Programs in Russia and Romania: Impact Evaluation Study.‖ Available at: http://www.cefir.ru/download.php?id=129 Betcherman, Gordon; Karina Olivas, and Amit Dar. ―Impacts of Active Labor Market Programs: New Evidence from Evaluations with Particular Attention to Developing and Transition Countries.‖ Social Protection Discussion Paper Series. No 0402. Social Protection Unit. Human Development Network. The World Bank. Available at: http://info.worldbank.org/etools/docs/library/251019/day6DiscussionPaperSeries0402April6Se1. pdf Bloom, Nicholas, Benn Eifert, Aprajit Mahajan, David McKenzie and John Roberts. June 21, 2012. ―Does management matter? Evidence from India.‖ Available at: http://www.stanford.edu/~nbloom/DMM.pdf Branstetter, Lee; Lowell Taylor, Ana Venancio. ―Do Entry Regulations Deter Entrepreneurship and Job Creation? Evidence from Recent Reforms in Portugal.‖ Working Paper 16473. National Bureau of Economic Research. Available at: http://www.nber.org/papers/w16473 39 Bruhn, Miriam. ―License to Sell: The Effect of Business Registration Reform on Entrepreneurial Activity in Mexico (February 1, 2008).‖ World Bank Policy Research Working Paper Series, Vol. pp. -, 2008. Available at SSRN: http://ssrn.com/abstract=1100353 Bruhn, Miriam and Bilal Zia. "Business and Financial Literacy for Young Entrepreneurs – Evidence from Bosnia-Herzegovina". Presentation for New Ideas in Business Growth: Financial Literacy, Firm Dynamics, and Entrepreneurial Environment. The World Bank. Available at: http://econ.worldbank.org/WBSITE/EXTERNAL/EXTDEC/EXTRESEARCH/EXTPROGRAMS/E XTFINRES/0,,contentMDK:22823089~pagePK:64168182~piPK:64168060~theSitePK:478060,0 0.html Bruhn, Miriam, Dean S. Karlan, and Antoinette Schoar. 2012. ―The Impact of Consulting Services on Small and Medium Enterprises: Evidence from a Randomized Trial in Mexico.‖ Yale Economics Department Working Paper No. 100. Bruhn, Miriam and Inessa Love. ―The Economic Impact of Banking the Unbanked: Evidence from Mexico.‖ Policy Research Working Paper 4981. The World Bank Development Research Group. June 2009. Available at: http://www- wds.worldbank.org/external/default/WDSContentServer/IW3P/IB/2009/06/30/000158349_20090 630083925/Rendered/PDF/WPS4981.pdf Card, David; Pablo Ibarraran, Ferinando Regalia, David Rosas and Yuri Soares. ―The Labor Market Impacts of Youth Training in the Dominican Republic: Evidence from a Randomized Evaluation.‖ NBER Working Paper No 12883. Issued in February 2007. Available at: http://www.nber.org/papers/w12883 Crepon, Bruno, Florencia Devoto, Esther Duflo and William Pariente. March 31, 2011. ―Impact of microcredit in rural areas of Morocco: Evidence from a Randomized Evaluation.‖ Working paper. Available at: http://economics.mit.edu/files/6659Cueto, Begona; and F. Mato. ―A non- experimental evaluation of training programmes: regional evidence for Spain.‖ The Annals of Regional Science. 2009-06-01. Springer Berlin / Heidelberg 0570-1864. pp 415-433. Volume 43. Issue 2. Available at: http://www.springerlink.com/content/80226112w215j776/ Datar, Gayatri & Del Carpio, Ximena V., 2009."Are irrigation rehabilitation projects good for poor farmers in Peru ?,"Policy Research Working Paper Series 5154, The World Bank. Available at: http://ideas.repec.org/p/wbk/wbrwps/5154.html Delajara, M., S. Freije and I. Soloaga. (2006) An Evaluation of Training for the Unemployed in Mexico. OVE/WP-09/06. Diaz, Juan and Miguel Jaramillo. ―An Evaluation of the Peruvian ―Youth Labor Training Program‖ Projoven.‖ Working Paper: OVE/WP-10/06. October, 2006. Available at: http://idbdocs.iadb.org/wsdocs/getdocument.aspx?docnum=907634 Fares, J. and Olga Puerto (2009), ―Towards Comprehensive Training,‖ SP Discussion Paper 0924 (2009), The World Bank, Washington, DC Fajnzylber, Pablo, William Maloney, & Gabriel V. Montes-Rojas, ―Does formality improve micro- firm performance? Evidence from the Brazilian SIMPLES program‖ Journal of Development Economics, 94 (2011) 262-276. 40 Ibarraran, Pablo and David Rosas Shady. ―Evaluating the impact of job training programmes in Latin America: evidence from IDB funded operations.‖ Journal of Development Effectiveness. Vol. 1, Issue 2. 2009. Available at: http://www.tandfonline.com/doi/abs/10.1080/19439340902918094#preview International Finance Corporation. ―IFC Africa Investment Climate: Liberia.‖ Internal IFC evaluation. 2011. International Finance Corporation. ―IFC Africa Investment Climate: Burkina Faso.‖ Internal IFC evaluation. 2011 International Finance Corporation. ―IFC Africa Investment Climate: Rwanda.‖ Internal IFC evaluation 2011. International Finance Corporation. ―IFC Africa Investment Climate: Sierra Leone.‖ Internal IFC evaluation 2011. International Finance Corporation. ―IFC Dialog Focus Group Note (Sri Lanka).‖ Internal IFC evaluation. 2011. International Finance Corporation. ―IFC Ghana Microfinance.‖ Internal IFC evaluation. 2011. International Finance Corporation. ―IFC Lima Tracer.‖ Internal IFC evaluation. 2011. International Finance Corporation. ―IFC Review of SEDF Phase II.‖ Internal IFC evaluation. 2011. International Finance Corporation. ―IFC Secured Transaction Evaluation.‖ Internal IFC evaluation. 2011. International Finance Corporation. ―Suape Port PPP.‖ Internal IFC evaluation. 2011 International Finance Corporation. ―IFC Techcombank SME Banking.‖ Internal IFC evaluation. 2009. Kellog, Clifton G. ―Evaluating poverty outreach of small business lending: a study of BRAC Bank, Bangladesh‖, Enterprise Development and Microfinance Vol. 20 No. 3, September 2009. Klonner, Stefan and Patrick Nolen. ―Does ICT Benefit the Poor? Evidence from South Africa‖ Working Paper. University of Essex. May 2008. Available at: http://privatewww.essex.ac.uk/~pjnolen/KlonnerNolenCellPhonesSouthAfrica.pdf Kondo, Toshio; Aniceto Orbeta, Clarence Dingcong and Christine Infantado. ―Impact of Microfinance on Rural Households in the Philippines‖ NONIE WORKING PAPER NO. 4 January 2008. Lokshin, Michael & Yemtsov, Ruslan, 2003. "Evaluating the impact of infrastructure rehabilitation projects on household welfare in rural Georgia," Policy Research Working Paper Series 3155, The World Bank. 41 Mu, Ren and Dominique van de Walle. ―Rural Roads and Local Market Development in Vietnam.‖ Policy Research Working Paper 4340. Impact Evaluation Series No. 18. Development Research Group. Human Development and Public Services Team. August 2007. Available: http://www- wds.worldbank.org/external/default/WDSContentServer/WDSP/IB/2009/08/05/000158349_2009 0805163649/Rendered/PDF/wps4340.pdf The Multilateral Investment Fund of the Inter-American Development Bank and The International Youth Foundation. Final Report of the entra21 Program Phase I: 2001-2007. September 2009. Available at: http://idbdocs.iadb.org/wsdocs/getdocument.aspx?docnum=2191107 Puerto, Olga. ―Labor Market Impact on Youth: A Meta-Analysis of the Youth Employment Inventory.‖ Background paper for the World Bank‘s 2007 Youth Employment Inventory. Social Protection and Labor Unit of the Human Development Network. The World Bank. Available at: http://siteresources.worldbank.org/EXTECAREGTOPYOUTH/Resources/YEI_MetaAnalysis_rev .0711.pdf Schoar, Antoinette, et. Al. ―Supervisory Skills Training in the Cambodian Garment Industry: A Randomized Impact Evaluation‖ ideas42, paper commissioned by IFC (2011). The World Bank. ―Issues in Estimating the Employment Generated by Energy Sector Activities.‖ Sustainable Energy Department. June 2011. Available at: http://siteresources.worldbank.org/EXTENERGY2/Resources/MeasuringEmploymentImpactofE nergySector.pdf 42