Policy Research Working Paper 9408 Small Business Training to Improve Management Practices in Developing Countries Reassessing the Evidence for “Training Doesn’t Work” David McKenzie Development Economics Development Research Group September 2020 Policy Research Working Paper 9408 Abstract Despite the popularity of business training among policy of this magnitude are too small for most experiments to makers, the use of business training has faced increasing detect statistically. Emerging evidence is provided on five skepticism. This is, in part, fueled by the fact that most of approaches for improving the effectiveness of traditional the first wave of randomized experiments in developing training by incorporating gender, kaizen methods, localiza- countries could not detect statistically significant impacts tion and mentoring, heuristics, and psychology. Training of training on firms’ profits or sales. This paper revisits programs that incorporate these elements appear to deliver and reassesses the evidence for whether small business improvements over traditional training programs on aver- training works, incorporating the results of more recent age, although with considerable variation. Given that studies. A meta-analysis of these estimates suggests that training delivers some benefits for firms, the challenge is training increases profits and sales on average by 5 to 10 then how to deliver a quality program on a cost-effective percent. The author argues that this is in line with what basis at a much larger scale. Three possible approaches to is optimistic to expect given the relatively short length of scaling up training are discussed: using the market, using most training programs, and the expected return on invest- technology, or targeting and funneling firms. ment from the cost of such training. However, impacts This paper is a product of the Development Research Group, Development Economics. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The author may be contacted at dmckenzie@worldbank.org. The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team Small Business Training to Improve Management Practices in Developing Countries: Reassessing the Evidence for “Training Doesn’t Work” # David McKenzie, World Bank Keywords: Business training; microenterprises; management practices; soft skills JEL Classification codes: O12, O17, L26, I26 # I thank Chris Woodruff for his collaboration on an earlier review of the business training literature; Tim Ogden for helpful discussions which helped prompt me to write this current review; and Stephen Anderson, Shawn Cole, Graeme Harris, Seema Jayachandran, Bill Maloney, Caio Piza, Simon Quinn, Petr Sedlacek, Bilal Zia, and an anonymous referee for helpful comments. 1. Introduction Business training programs to help new and existing small firms improve their business practices are offered by many governments, microfinance organizations, NGOs, and private providers. For example: the International Labour Organization’s Start and Improve Your Business (SIYB) program has a network of over 65,000 trainers and estimates that at least 15 million clients have been trained (van Lieshout and Mehtha, 2017); CEFE International reports having more than 20,000 trainers qualified in its methodology in 140 countries, and having trained 20 million participants 1; and India’s National Institute for Entrepreneurship and Small Business Development reports having provided 44,035 different training programs to more than 1.1 million people. 2 These are all cumulative numbers since the programs started, and although information is fragmented and incomplete, my rough estimate is that at least $1 billion is spent annually training at least 4 million to 5 million potential and existing entrepreneurs in developing countries. 3 What explains the enormous popularity of business training programs? Many small firms in developing countries are not using basic business practices like keeping records, advertising their goods, budgeting, and planning. Evidence from a wide range of countries shows that better management does matter for firm productivity and growth among small and medium enterprises (SMEs) (Bloom and van Reenen, 2010; Bloom et al. 2013), and that, even among the self- employed or firms with just a few employees, firms using better business practices are more profitable and grow faster over time (McKenzie and Woodruff, 2017). It is therefore natural for policy makers to wish to train firms to use these beneficial business practices. In addition to these evidential reasons, I believe that there are also idealistic and political economy concerns that are important in understanding why business training programs are so beloved by governments, NGOs and donors. Small businesses are often romanticized in popular discussion (Ozimek, 2013) and such firms are an important source of income for the poor. Helping lots of small firms is then often politically popular. But different forms of support to small firms vary in 1 Source: https://cefe.net/about/ [accessed June 2, 2020]. 2 Source: https://www.niesbud.nic.in/ [accessed June 2, 2020]. 3 This is a rough estimate with considerable uncertainty. Van Lieshout and Mehtha (2017) report the ILO’s SIYB program training 10.5 million clients over the 5 years from 2011-15, of which 10.0 million were in China. As we will discuss later in the paper, a typical training program costs $150 to $1500 to provide. 2 this popularity. The old proverb 4 “give a man a fish and you feed him for a day, teach a man to fish and you feed him for a lifetime” plays a powerful role in the view among many policy makers that it is better to spend money teaching skills than to directly support firm owners by simply giving them cash. This is regardless of evidence showing that small firms can have large returns to small cash grants (de Mel et al, 2008). Moreover, from a practical political economy side, providing large numbers of training sessions provides a visible sign of support for small businesses, can be relatively easy to contract and organize, and may be less susceptible to monitoring and corruption concerns that some other forms of small business support. However, more recently there has been a bit of backlash and more skepticism about business training in both the academic and policy communities. One reason for this has been a reaction to the first wave of randomized experiments measuring the impacts of business training in developing countries. McKenzie and Woodruff (2014) critically reviewed thirteen of these experiments, and report that only two studies found significant impacts on firm profitability, with most studies finding statistically insignificant impacts. This lack of significance has been interpreted by many readers as showing that traditional business training does not work. For example, Fox and Thomas (2016, p.i33) cite this work to conclude that “what is clear is that projects to graduate household enterprises into small business through entrepreneurship training are expensive and do not seem to pay off”, while Brooks et al. (2018, p.197) cite it to write “where formal business classes have been offered to entrepreneurs, they have had limited impact”. Moreover, to the extent that there are benefits, a further concern of critics is that these benefits may be largely private benefits that accrue to the participating firms, and may come from stealing away sales from other (non-trained) firms. This raises the question of why the government should spend public money on helping private firms. More broadly still, the current mix of funding and policy attention devoted to small firms has been questioned, with arguments that public policy should pay more attention in developing countries to supporting the creation and growth of formal jobs in large firms (e.g. Ciani et al, 2020). My goals in this paper are to revisit and reassess the evidence on small business training in light of new literature that has emerged since the review of McKenzie and Woodruff (2014). I start by 4 Interestingly, the origins of this proverb are highly disputed, but the general principle of alleviating poverty through facilitating self-sufficiency dates back to at least the 12th century (https://quoteinvestigator.com/2015/08/28/fish/). 3 reviewing what a typical business training program entails, and the challenges of learning whether such programs work or not. I discuss what is realistic to expect from business training given the length of such programs and what we know about the returns to education and capital from other settings. A general point from this is that, even if training works as well as formal education, the gains from interventions currently in use may be difficult to detect. I then summarize in a couple of simple figures the range of estimates of the impacts of business training on business profits and sales. The results show very wide confidence intervals from many studies, which I use to re- emphasize the point that not finding a significant effect of training is not the same as finding that training has no effect. However, a meta-analysis of what we learn from all these studies combined suggests an average impact of 10 percent on profits (95 percent confidence interval (C.I.) of +4, +16) and 5 percent on sales (95% C.I.: 0.2, 9.2). That is, that overall training does deliver some benefit to firms, just a benefit that is too small for most experiments to have detected. I then turn to discussing two areas where I see a lot of innovation and promise in efforts to improve business training. The first is in considering alternatives to traditional classroom-based business training that aims to teach a broad range of basic business practices like accounting, marketing, human resource management, inventory management and forward planning. These alternatives use psychology, heuristics, incorporate the role of gender, use peers or mentors, aim to customize practices to local context, or use more technical lean manufacturing approaches all in efforts to help firms improve. A meta-analysis of these approaches finds average impacts of 15 percent on profits (95% C.I.: +8, +22) and 11 percent on sales (95% C.I.: +3, +20), suggesting an improvement over traditional training. However, to date most training studies have been conducted on relatively small scales under close researcher control. The second issue I consider is then how to deliver business training at scale, so that it can reach tens of thousands or even millions of firms. I consider the roles of using the market, using online technologies, and using filtering or funnel approaches to offer more services to the firms that will benefit most from them. I then conclude by highlighting what I see as the pressing open research and policy questions given the existing state of evidence. My focus is on business training programs for the typical micro and small firms in developing countries. Many of the lessons will also be relevant for small firms in developed countries, for which there are far fewer experimental evaluations (Fairlie et al. 2015 is one exception). I do not 4 cover bundled interventions, in which business training is combined with other interventions such as microfinance, grants, or other programs so that the effect of training per se cannot be measured. I also do not consider training designed for equity-seeking start-ups, such as investment readiness programs (e.g. Cusolito et al, forthcoming), training that takes place in accelerators or incubators (e.g. Gonzalez-Uribe and Leatherbee, 2018), or consulting interventions. 5 My focus is on training designed for improving business practices, and so I also exclude vocational training programs aimed at improving the sector-specific technical skills of entrepreneurs (see Blattman and Ralston, 2017 for a recent review). 2. What does a typical business training program look like, what should we expect it to do, and how can we learn whether it works? Although there are a wide variety of different business training programs and approaches, a typical program involves a trainer teaching a group of 15 to 40 participants in a classroom setting, over a period of 3 to 12 days. Courses focused on potential entrepreneurs looking to start a business focus on topics like generating your business idea, developing a business plan, permits, costing, pricing and budgeting. Courses focused on existing firms looking to grow cover record keeping and accounting, marketing, human resources and hiring workers, stock control and inventory management, planning, and operations management. Some of the most well-known courses are the SIYB program of the ILO (Majurin, 2014), CEFE International’s range of courses, Freedom from Hunger’s courses for microfinance clients, and the Business Edge program formerly offered by the IFC. 6 Most of these training programs train a set of master trainers, who in turn train a network of trainers in different countries. The course materials are typically translated and adapted to local contexts, so that the examples given are relatable to participants. While courses are typically classroom based, with the trainer standing up in front of a class and explaining concepts, these courses also 5 See https://www.povertyactionlab.org/policy-insight/supporting-firm-growth-through-consulting-and-business- training for a recent review of consulting interventions. 6 IFC no longer offers the Business Edge program. Instead, it has recently launched the Grow, Learn, Connect skills development program hosted on https://www.growlearnconnect.org/. The program includes a set of developed standards (Principles for Learning and Guide to Training), certifications, capacity-building courses for training professionals and a digital platform. While the Business Edge program was designed to improve management capacity and business performance of MSMEs directly, the focus of the new program is solely on training professionals. 5 try to incorporate active learning through having participants do exercises or games to help explain key concepts, and tasks which have them apply the content to their own businesses. McKenzie and Woodruff (2014) report that the cost of the training programs studied in the first wave of randomized experiments varied from as low as $21 for a course offered by local instructors in local schools in the Dominican Republic by Drexler et al. (2014), to $740 for a three week training course for metalwork entrepreneurs in Ghana by Mano et al. (2012). Van Lieshout and Mehtha (2017) report costs for offering a 5-7 day SIYB course in 18 different countries, which range from $400 to $12,242 for a class of 20, averaging $3,537 or $177 per participant. In many countries, and most randomized experiments, training is offered to firms for free or for a token cost – van Lieshout and Mehtha (2017) report a median contribution by participants of 10 percent of the cost. This large variability in costs reflects differences in whether instructors are specialist trainers or NGO staff, whether venues need to be hired or classes can be held in schools or halls without charge, on transport costs for getting instructors to remote areas, and on the scale of trainings being offered. Governments attempting to procure training services and provide them nationwide are largely to face much larger costs. For example, the Nigerian GEM project paid approximately $2,000 per firm for a blend of 5 online modules followed by 12 days of in-person Business Edge business training (Anderson and McKenzie, 2020). Higuchi et al. (2019) also report a cost of $2,000 per firm for their kaizen training program (discussed below). What constitutes success from business training and what should we expect training to do? Standard business training programs aim to teach participants a range of better business practices. The first thing we would like to see from training is therefore likely to be that the knowledge of participants increases, and that they are more likely to use these business practices in their businesses. However, while the state might have a direct interest in getting firms to use some of these practices (e.g. record-keeping, caring for workers, licensing) irrespective of their impact on firm growth, ultimately firms and policy makers will mostly want to judge the success of business training on whether it helps firms to survive, sell more, and become more profitable and productive. In particular, in thinking about the cost-benefit of firms or governments investing in business training, we will want to know whether firms’ profits increase by enough to cover the costs incurred. 6 There are multiple potential channels through which business training can help firms grow and become more profitable. Better business practices may enable firm owners to produce their goods more efficiently, lowering expenses (e.g. through less wastage, negotiating better input prices) and therefore increasing profit margins on each item sold. Better marketing practices can help firms attract new customers and sell more, while customer research, costing, and planning can help firms identify new products to introduce that may sell better. Better managed firms may also be more attractive to potential lenders and investors, allowing firms access to capital that they can use to grow, and well-managed firms may also be better able to compete for government contracts. Training may also change the aspirations and mindset of the entrepreneur. Finally, this combination of effects may enable firm owners to better react to shocks to the business, making it more likely that the business will survive downturns. However, several conditions need to be in place for this promise to be realized. If the binding constraint to firm growth is lack of finance, lack of customer demand, or lack of another key input, then unless business training can alleviate this constraint, it may not help firms to grow even if they implement all the recommended practices. It may also take time for impacts to appear, or conversely short-term impacts may fade quickly as firm owners forget what is learned or as the specific knowledge depreciates in value. What I see as the biggest issue for considering how much impact to expect is the short length of most business training programs. If training consists of just five days, we need to be realistic about how much of a change in the business can be expected. One benchmark for comparison is to compare to the returns to formal education. A full year of formal education in the average developing country has a return of 7.6 percent (Peet et al., 2015). Suppose that business training is twice as effective as regular schooling. Even then, since 5 days is only about 1/30th of a school year, we might expect returns from 5 days of training to be only 0.5%. Or even if business training is 20 times as effective as regular education in increasing incomes, we should still only expect a 5 percent increase in profits. An alternative is to take a return on investment approach. Consider a firm making an investment of $177, the cost of an average SIYB course. De Mel et al. (2008) report that microenterprises have the very high return on capital from $100-$200 investments of 4.6-5.3 percent per month, much higher than market interest rates. Even with such a return, firms would earn an additional 7 $8-$9 per month from the investment. For a small firm earning $100 a month, this would be a 8-9 percent increase in profits. If we instead use a market interest rate of approximately 2 percent per month, then firms would earn $3.50 more per month from training, which would be a 3.5 percent increase in profits for a firm earning $100 a month. If firms are more profitable to start with (small rather than micro), the percent increase in profits needed for training to pay for itself becomes even smaller. From either this return to education or this return to investment approach, we see that an opportunistic estimate of the increase in profits we should see from training is in the 4 to 5 percent range, and the return could be smaller and still compare favorably with regular education and pass a cost-benefit calculation for relatively profitable firms. This will present a challenge for detecting and measuring the impacts of training, since most studies do not have enough statistical power to detect impacts of such a magnitude. How can we learn whether or not business training works? The approach that many training providers use to measure impact is to simply compare before and after outcomes for participants. For example, as of June 2, 2020, CEFE International’s website claims an impact of 18 million jobs created based on interviewing 459 participants who had participated in training 2 years earlier, and asking them how many additional workers they had hired since training, and then multiplying this average of 1.4 by their total number of participants. 7 Organizations often complement this by case studies of their most inspiring success stories – for example, the ILO’s SIYB website features 10 inspiring stories. 8 Such approaches almost certainly overstate the benefits from training. The key challenge to learning the impact of business training is being able to compare outcomes after training to a reliable counterfactual of what would have happened without training. The problem with before-after comparisons is that individuals may decide to take part in business training when they are planning on starting a business or expanding it anyway. Even if they do not receive training, they might still grow their businesses. De Mel et al. (2014) provides an example of this from the SIYB training in Sri Lanka. They randomly allocated women without businesses 7 https://cefe.net/cefe-impact/ [accessed June 2, 2020]. 8 https://www.ilo.org/empent/areas/start-and-improve-your-business/WCMS_729986/lang--en/index.htm [accessed June 2, 2020]. 8 to be invited to receive training or not. Two years later, 68.8 percent of the control group of firms who had not been offered training had still managed to start a business. Such is the volatility of small firm profits, and the number of other shocks affecting the business, that it may also be hard for an individual firm that has taken training to assess how much it has helped them. So this problem is unlikely to be able to be solved by asking firms what they would have done without training. If before-after comparisons are no good, then perhaps we can compare changes in outcomes for firms taking part in training to those not taking part in training? This approach will also be problematic in general, if firms get to self-select into whether or not they participate in training. For example, it may be the most ambitious, growth-oriented, firms that want to take part in training, while firms that are happy with their status quo may not take part. This is the advantage of the use of randomized experiments in assessing impacts. These studies take a group of potential entrepreneurs or existing firms, and then randomly allocate some firms to be offered training and compare them to a control group of firms that are not offered the training. Comparing outcomes for the treatment group which receives training to the control group that does not then enables an estimate of how much difference training has made. In the above Sri Lankan training example, 70.4 percent of firms offered training had started a business after two years, which can be compared to the 68.6 percent in the control group to give an estimate that training only increased the likelihood of starting a business by 1.6 percentage points. Although using randomized experiments helps overcome many of the problems in assessing impact, there are several important caveats and technical issues. Many experiments have used sample sizes of 200 to 300 firms, only measure outcomes within a relatively short time frame (e.g. 1 year) after training, and suffer from problems of incomplete take-up (not all those offered training participate) and survey attrition (post-training outcomes are not always available for all firms). Measuring business profits and sales in small firms is difficult, and responses often have substantial noise (de Mel et al, ; Anderson et al, 2019). The main consequence of these factors is that many experiments have far less statistical power (less precision) in measuring the outcomes of business training than we would like. McKenzie and Woodruff (2014) discuss many of these issues in more depth, and show that almost all of the first wave of randomized trials lacked power 9 to detect a 25 percent increase in profits. We see this more clearly in the next section, which shows confidence intervals for the treatment effects estimated from different training programs. 3) What does the experimental evidence of the impacts of business training show? Most studies of training show statistically significant impacts of training on business practices. Comparing these impacts across studies is complicated by different studies measuring different practices, and reporting treatment effects in different units. Studies which measure a range of different practices and then estimate the impact of training on the proportion of practices that firms use typically find effect sizes in the range of 0.02 to 0.07 (e.g. De Mel et al. (2014), Campos et al. (2017), Ubfal et al. (2019), McKenzie and Puerto (2020), Dalton et al. (2020)). That is, for every 20 practices that business training attempts to teach firms to do, on average firms invited to training only implement one additional practice. McKenzie and Woodruff (2017) discuss this as one way of reconciling the strong association between business practices and firm profits and sales with the lack of statistical significance of many of the early studies of training on these firm outcomes – business practices matter, but short training sessions often do not lead to big changes in these practices. Of course, training may change practices that researchers are unable to measure, provide useful information about accessing finance, or change entrepreneurial attitudes. What matters is how these changes translate into impacts on profits and sales. Measuring the impacts of training on the sales and profits of small firms is not straightforward for several important reasons. The first is that not everyone who is invited to training, attends. For this reason, most studies report what is called the intent-to-treat (ITT) effect, which measures the impact of being invited to training, regardless of whether an owner attends all sessions. But this incomplete take-up of the training lowers statistical power, making it harder to detect training impacts. Second, small firms in developing countries die at an average rate of 8.2 percent per year, so that half of the firms operating at a given point in time will die within the next six years (McKenzie and Paffhausen, 2019). Some researchers code firms that die as having zero profits, while others report results conditional on survival, which can introduce the potential for bias if training changes how many or which firms survive. 9 Third, one consequence of poor business practices is that many small 9 This can be an even larger issue for programs focused on individuals trying to start businesses, especially youth programs, since few of those trained may be operating a firm, and training may affect which individuals start firms. 10 firms do not keep good profit and sales records, and, in some contexts, informal firms may be reluctant to reveal profit and sales information due to concerns about taxation or theft. The combination of survey attrition and firm death can mean that in some studies only one-half of the original sample of firms end up having their outcomes compared, and then researchers need to carefully use different methods to argue why this does not cause bias. Finally, there is tremendous heterogeneity in the outcomes of small firms in developing countries, and profits and sales can vary a lot from one month to the next (Fafchamps et al, 2012). The result is that the average profits and sales of a group of firms can be heavily influenced by one or two outliers and by the top tail of the distribution. The measured percent change in profits or sales can be quite sensitive to choices researchers make of how to deal with these outliers (e.g. transforming profits with a logarithmic or inverse hyperbolic sine transformation, winsorizing profits to reduce the influence of the top tail, etc.). Having noted these caveats, I take the reported ITT treatment impacts from each study at face value, and to enable comparability across studies, transform impacts into percentage changes. It seems more likely to think that training will have the same percentage impact on different firms than the same level effect (e.g. that it increases profits by 5% for all firms, rather than that it increases profits by $100 for all firms). The downside of using percentage changes is that treatment impacts that are large in percentage terms may be small in absolute terms for very small firms: a $10 a month increase in profits for a subsistence firm earning $1 per day is a large percentage increase, but still take a lot longer to recoup the costs of training than a $50 a month increase in profits for a small firm earning $1,000 a month. Figures 1 and 2 then plot the estimated impact of different studies of traditional business training on firm profits and firm sales respectively, with studies ordered by the year in which training occurred. The studies included in the review of McKenzie and Woodruff (2014) were those where training had begun by 2009. We see that few of these early studies had statistically significant impacts, with some studies even having negative point estimates of the impacts on profits and sales. However, the confidence intervals for these impacts are typically quite wide, with most including the possibility that training increased profits and sales by 25 percent, but also the For example, Alaref et al. (2020) find only 5 percent of their sample of Tunisian youth who went through an entrepreneurship program in the final year of university are operating a business four years later. 11 possibility that profits and sales had fallen by 10 or 20 percent. That is, lack of a statistically significant effect does not mean that these studies show that training has no effect, only that they cannot detect what the effect is. Many of the subsequent studies have attempted to improve the precision of these estimates by training a larger sample of firms, screening the sample to reduce heterogeneity, and pooling together multiple rounds of follow-up surveys. The result is that confidence intervals are narrower for some of these more recent studies. Most of these recent studies have positive point estimates, but many also have confidence intervals that contain zero. Most studies therefore cannot reject the hypothesis that training did not improve profits or sales, but they almost all allow for the increase in profits of 5 percent that I noted in the previous section as realistic based on both course length and return on investment. Using a random-effects meta-analysis model, 10 we can combine the results of all these studies to get an overall estimate of the effect on profits of 10.1 percent (95% C.I.: +4.1, +16.1), and on sales of 4.7 percent (95% C.I.: +0.2, +9.2). That is, looking at the totality of evidence from all of these studies, one concludes that business training has improved business outcomes, just not be enough to be detectable in most individual studies. The red line shows these average effects, which lie inside the confidence intervals for almost every study. Note also that the meta-analysis puts the largest weights mostly on studies that have happened more recently, while some of the older studies that had small samples and very wide confidence intervals get very little weight. 10 The meta-analysis is carried out using Stata’s meta forestplot command. This takes as inputs the point estimate and standard error from each business training study. The study-specific effect is modeled as = + + , where it is assumed that the observed study-specific effect sizes are sampled from a population of effect sizes with mean θ and u and ε are independent with ~�0, 2 � and ~(0, 2 ). Maximum-likelihood is then used to estimate θ as a weighted average of the different effect sizes, where the weights are estimated as 1/� �2 + ̂ 2 �. The result is that higher weight is given to studies that have smaller standard errors. 12 Figure 1: Estimates of the Impact of Business Training on Firm Profits Notes: Effect size is percentage change in profits. Number Trained is number of firms invited to training. Green diamond shows random effects meta-analysis estimate, and red line shows this estimate. Weight is weight random effects meta-analysis gives to study, with studies with smaller standard errors given larger weight. Where impacts were available over multiple time horizons, the longest time horizon is chosen. 13 Figure 2: Estimates of the Impact of Business Training on Firm Sales Notes: Effect size is percentage change in profits. Number Trained is number of firms invited to training. Green diamond shows random effects meta-analysis estimate, and red line shows this estimate. Weight is weight random effects meta-analysis gives to study, with studies with smaller standard errors given larger weight. Where impacts were available over multiple time horizons, the longest time horizon is chosen. 14 4) What are some of the alternatives to traditional business training, and the evidence on their effectiveness? Rather than just asking does business training work or not, an alternative is to ask what can be done to make training work better. One possibility is to expand and customize what is taught. These approaches still seek to teach a reasonably comprehensive set of business practices, but either add additional content, or change how it is taught. A second possibility is to change what is taught by moving away from trying to teach a broad set of business practices and instead providing simple heuristics, or using psychology to develop alternate dimensions of business skills. We discuss these possibilities in turn, and summarize their impacts on profits and sales in Figures 3 and 4. Gender-oriented training for women A first way of broadening the content is to add material that is designed to help overcome gender constraints. Some of the literature on standard business training suggests that it may be less effective for women than men, either because women work in sectors with little scope for growth, or because they face other constraints. Some training programs aim to explicitly address these additional barriers, either by helping women enter new sectors, or teaching them how to better deal with dividing household and business tasks, bargain better, work together with other women, and overcome stereotypes. An example of a training program with a gender focus is the ILO’s Gender and Enterprise Together (GET Ahead) program, which combines standard topics like recordkeeping, separating business and household finances, costing and pricing, etc. with topics on gender. This program has been evaluated in two randomized experiments. In Vietnam, Bulte et al. (2016) work with 4,041 women in 187 credit centers. They find business training increased business knowledge: at baseline the control group got an average of 8.9 out of 17 questions correct, which increased by 2.2 to 2.4 questions compared to the control group after training. Business practices also significantly improved, although the magnitude is not easily comparable to other studies. The effects on business profits and sales are much noisier, with very wide confidence intervals. McKenzie and Puerto (2020) work with 3,537 firms in 157 rural markets in Kenya. They find training led firms to implement 5 to 7 percentage points more business practices, and that firm profits increased 5 15 percent over one year (95% C.I.: -4.0, +14.8) and 15.4 percent over three years (95% C.I.: +3.4, +27.3). Moreover, they find that this growth for the trained women does not just come from stealing sales from women in the control group, but instead total market sales grew. One channel appears to be innovation as trained firms introduce new varieties. Both of these studies suggest that the impacts of training on firm profitability can take time to materialize for women, with one reason in Kenya being that women lack access to finance, and so had to gradually grow their firms by re-investing profits and building up their inventory levels over time. But while they show this program can lead to firm growth, since neither study compares it with a training program without an explicit gender focus, it is hard to know how important the gender focus was in generating the results. Indeed, McKenzie and Puerto (2020) report that they do not see significant impacts on gender attitudes, self-efficacy, or working together with other women, and associated qualitative work does not find women reporting the gender component as especially useful. The magnitudes of impacts found on business practices and profits are in line with studies of traditional training, and so it may just be the larger sample sizes and careful measurement that enabled significant impacts to be detected in these studies. Attempts to make training more useful for women may also want to alter not just the content of the training, but also the way women are invited, and the complementary services provided. Field et al. (2016) find that women who are invited to attend training with a friend are more likely to borrow and may lead women to set different goals for themselves during training. Some training programs also offer childcare and other related services to help women overcome constraints that would otherwise prevent them from taking part in training. Kaizen approaches A second way of broadening content is to add material that focuses on production and quality management, typically tailored to a single manufacturing industry. Several studies have taught the Japanese concept of kaizen, or continuous improvement, to firms organized in industry clusters. This includes examining workflow and bottlenecks, the importance of routine machine maintenance, and techniques such as 5S, which seek to reduce waste, and ensure the workspace is clean and uncluttered. The first experiment to test this approach is Mano et al. (2012), who added a module on this approach to two modules based around the SIYB approach, working with 167 small metalwork firms in Ghana. They find a year later that training has increased survival by 8 to 16 9 percentage points, and that impacts on profits and sales are imprecisely measured (a 95 percent confidence interval for the impact on profits is -66%, +115%). This approach has been followed by experiments among 316 steel construction and knitwear firms in Vietnam (Higuchi et al, 2015), and for 113 garment firms in Tanzania (Higuchi et al, 2019). In both cases the researchers split the samples into four groups, comparing those who got classroom training only to those who got training onsite, or a combination. The result is that the number of firms getting any particular training combination is small – approximately 30 in the Tanzania example. Neither study estimates impacts on profits, but they both show firms adopting more management practices, and that value-added improves after some time. A longer-term follow-up of the Vietnam sample by Higuchi et al. (2017) finds firms assigned to training were 17 percentage points more likely to still be in business five years after training. The Tanzania study also finds large increases in sales, with sales 90 percent higher after three years for the group getting both classroom and onsite training (95% C.I.: +33, +148). These are extremely large effects, particularly because Higuchi et al. (2015) note in the Vietnam study that the training emphasizes cost reduction, not revenue generation. My view is that these results suggest potential in this kaizen approach, but that I would like to see studies on much larger sample sizes to feel more confident in these results. Moreover, it would be good to benchmark it against a standard training program, to measure how much additional benefit the kaizen content provides. Local customization and the use of peers or mentors Most global training providers already do some customization of content when they introduce it into individual countries. But there are limits to how much customization takes place, and once training materials are produced, the content may not quickly reflect sector-specific needs, or changes occurring in the local economy. Training may be more relevant and useful to small firms if it can be better adapted to local conditions. One approach to doing this was tested by Dalton et al. (2020). They conduct qualitative interviews with local retailers in Indonesia, and couple this with regressions of the associations between different business practices with profits and sales. Using this information, they put together a handbook that curates local best practices, corrects common misperceptions about implementing these practices, and provides specific examples of how firms have gone about using a particular practice in the local context. They do not find any significant impacts of the handbook by itself, 17 but when it is coupled with two thirty-minute visits to help the firms implement these practices (and a video showing role models talking about using them that appears to add no additional benefit beyond the in-person visit), they find that approximately a year later, firms have increased business practices by 5 to 6 percentage points, firm profits have increased by 21 percent (95% C.I.: +2.8, +39.9) and sales have increased by 16 percent (95% C.I.: +2.1, +30.2). The cost was approximately $150 per firm, which is similar to many 5-day training programs, and the study cannot say whether impacts would have been any different if a standard training program was used. The main potential of this approach is that a lot of the costs were fixed costs of developing the handbook, and the in- person assistance visits only cost $25 per firm. Since classroom training does not occur, delivering handbooks accompanied by a couple of short assistance visits could be a cost-effective approach for improving business practices at a larger scale. In addition to helping to customize training, getting business owners to meet with peers can help in diffusing business information, and may aid in creating business partnerships (Fafchamps and Quinn 2016; Cai and Szeidl, 2018). However, only certain types of business practices may diffuse this way, as firms can be reluctant to share information that affects their competitiveness. Peer learning may not happen automatically, and training may help firms learn how to better communicate with one another. Dimitriadis and Koning (2020) conducted an experiment in Togo, in which entrepreneurs were given a 2-hour communication training to help them better interact with peers, finding that this led to more information exchange and short-term performance gains. The use of a role model who explains how the knowledge being provided in the course has helped them in their business, as well as who potentially gives out some practical actionable information (e.g. where to apply for financing) can potentially enhance the effectiveness of training. The above Indonesian study tried this via videos. LaFortune et al. (2018) report on an experiment in Chile which had successful alums appear in person to give a motivational speech during training. They find some evidence that this increased the impact of training on profits 9 to 12 months later. However, the use of role models does not appear to improve the business practices used, but instead may work through changes in attitudes and in investment behavior. Another way of customizing advice to meet the needs of small businesses is through mentors. One use of mentors has been to have local mentors (or, in some cases, instructors) follow-up after classroom training with several in-person visits intended to reinforce the content taught in training 18 and help firms with any implementation issues. This approach was tested by Drexler et al. (2014) in the Dominican Republic, Valdivia (2015) in Peru, McKenzie and Puerto (2020) in Kenya, and Gine and Mansuri (2020) in Pakistan. None of these studies finds any sustained impact of these follow-up visits. Valdivia’s results are the most suggestive of an effect, with sales impacts after 7 to 10 months approximately twice as large as for training alone. However, he cannot reject equality of the training only and training plus technical assistance treatments, and after two years, both treatments have similar point estimates. A couple of caveats here are that some of these studies just use instructors to do a handful of in-person follow-ups, which can be very different from an individualized mentoring experience, and they may not have statistical power to detect impacts. The other approach to using mentors is to use them as a substitute for, instead of complement to, classroom-based training. Brooks et al. (2018) is an example of this approach. They selected the more profitable business owners from among their sample to act as mentors for a nominal payment ($9.83). Compared to business training, they find that this mentoring led to greater short-term improvements in profits, but this impact appears to have dissipated over time, and is much smaller and not significant at 12 and 17 months. They argue that the mentors were useful at solving specific problems facing firm owners (e.g. where to get a particular input), but did not result in lasting benefits for the treated businesses. There are many implementation details that are likely to affect the effectiveness of mentoring. Examples include the experience and skills of the mentors, how relatable they are to the entrepreneurs they are helping, the length and frequency of their interactions with firms, whether mentors are compensated for their time or do this as volunteers. My hypothesis is that mentors may be more helpful for firms looking to innovate or expand into new markets, where their local networks may not be good substitutes, but more research is needed on both what types of firms to offer mentors to, as well as how to structure the mentoring. Anderson et al. (2020) provides some evidence for this idea that external expertise can help. They conduct an experiment with 930 Ugandan entrepreneurs, and match the treatment group with volunteer “coaches” in other countries who work virtually with them over 2 to 4 months to help the firm improve. They find that over a 19 two-year period those firms matched with marketing experts experienced a 36 percent increase in profits (95% C.I.: -2, +74), which they suggest comes through more product differentiation. 11 Matching individual firms with a mentor or coach who individually helps them improve business practices looks a lot like business consulting. Several experiments have found that intensive individualized consulting can help firms improve management and grow (Bloom et al, 2013; Bruhn et al, 2018; Iacovone et al; 2020; Anderson and McKenzie, 2020). While I have tried to focus this review on business training rather than consulting, I acknowledge the dividing line between consulting and training is fuzzy. While one division is between cheaper, group-based training, and very expensive one-on-one consulting, some recent studies combine elements of both approaches. Iacovone et al. (2020) show that the costs of consulting can be lowered by having a consultant work with small groups of firms to improve their management, in a similar way to agricultural extension and group trainings; while Anderson et al.’s (2020) one-on-one coaching or Brooks et al.’s (2018) mentoring are cheaper ways of providing individual consulting. Simplifying training through heuristics and rules-of-thumb Standard training programs typically attempt to teach a broad range of business practices, and may be overly complex for less-educated individuals running subsistence microenterprises in developing countries. An alternative is to change what is taught, by simplifying, and focusing on some heuristic guidelines and rules-of-thumb. This approach was tested by Drexler et al. (2014) in an experiment with 1,193 microenterprises in the Dominican Republic. Their rules-of-thumb training focused on techniques to separate household and business finances, and was compared to a control group and a group that received more standard accounting training. They cannot reject equality of impacts of the two treatments for their full sample, but find that the rules-of-thumb training worked better than accounting training in improving business practices and revenues for firm owners that had the lowest levels of business practices to begin with, and for those who were less educated. This sounds intuitively appealing, and this study was novel among the first batch of training experiments in testing an alternative to traditional training. However, the results are not that strong. The authors do not report impacts on firm profits, have missing data on sales for more than half 11 I have not included this study in the meta-analysis, since it is perhaps more akin to one-on-one consulting than training. Moreover, the authors randomize firms to three types of external experts, and do not report the pooled effect. 20 their sample, and the impact of their rules-of-thumb training on sales in an average week is relatively small (4.9 percent) and not statistically significant (95% C.I.: -14.6, +24.4). It is therefore pleasing to see a recent study replicate this approach. Arráiz et al. (2019) tested a short (4 hour) heuristic training program again focused around rules-of-thumb for finances against an accounting and finance training program among 2,408 microenterprises in Ecuador. They find the heuristic-based approach increased daily profits a year later by 8.1 percent (95% C.I.: +0.7, +16.0), and daily sales by a similar amount. The program seems to do better for women, and for those with lower cognitive scores. The magnitudes of the accounting and finance training are smaller, but they cannot reject that the two forms of training have equal impact on their main sales and profits indices. Another example of providing simplified rules to help improve business practices comes from Beaman et al. (2014) in Kenya, who found that very small retailers would lose sales because they did not have change for larger bills offered by customers. They find that informing firms of the importance of keeping correct change reduces change-outs, and led to increases in profits within the three-month period they monitor. Taken together, these studies point to the importance of developing simplified rules that can help the smallest businesses and least-educated entrepreneurs have slightly better business practices. However, several caveats remain about this approach. First, all of the studies only follow firms over relatively short horizons (at most one year), and it is unclear whether firm owners stop using these heuristics over time, or whether they become ingrained habits. Second, to date these heuristics have largely focused on financial management, and it is less clear whether relevant heuristics can be also used for other types of important business practices like marketing, stock control, and forward planning. Using psychology to dimension alternative dimensions of entrepreneurial skills Traditional business training programs have emphasized “hard” skills such as learning to keep accounts, put together a budget, or implement a marketing campaign. An alternative approach that has gained growing attention is to use lessons from psychology to develop the “soft” skills associated with successful entrepreneurship. One prominent example is personal initiative training. This aims to develop key behaviors associated with a proactive entrepreneurial mindset, such as constantly searching for new opportunities, being self-starting, learning from errors and feedback 21 to overcome obstacles, and thinking of ways to differentiate oneself from other businesses. For example, a training exercise involves entrepreneurs thinking through their previous day in the business, and asking what they can do so that tomorrow is an improvement and does not look the same as every other day. A pilot experiment of this idea was tested by Glaub et al. (2014) on 109 small businesses in Uganda, with a large, but imprecise, impact on sales one year later. Campos et al. (2017) then compared personal initiative training to the Business Edge training program in a sample of 1,500 firms in Togo. Both groups got 36 hours of classroom training, followed by a trainer visiting each business for 3 hours, once per month, for the next four months. The cost was approximately $750 per firm. They find that the personal initiative training resulted in a significantly larger improvement in business profits over the next two and a half years (a 29.9 percent increase, 95% C.I.: +15.4, +44.4) than the traditional training (11.2 percent increase, 95% C.I.: -2.7, +25.1). Moreover, the training showed positive impacts for both male and female entrepreneurs, and was effective for both more and less educated women (Campos et al., 2018). These experiments show the promise of personal initiative training to help small businesses innovate and grow. However, the quality of the trainers may matter even more for psychology- based training programs than for hard skills training. Alibhai et al. (2019) report on an experiment with 2,001 women in Ethiopia, where personal initiative training was taught through a government program at a lot lower cost (only $30 per firm), with no significant impacts on either sales or profits after 1.5 years (the point estimate on profits is a 1.2 percent increase, 95% C.I.: -11.0, +13.3). They find the change in personal initiative of entrepreneurs is strongly correlated with trainer characteristics, and in particular, with whether the trainer has previous business experience. In a second experiment with a different mindset training program (StartUp! and ReachUp!) in the same country with different trainers, they do find more impact on profits after one and two years, of approximately 20 percent. Ubfal et al. (2019) also test personal initiative training in a sample of 945 entrepreneurs in Jamaica, and test whether the impacts differ if training is a combination of personal initiative and traditional training, compared to soft skills only. They find the soft skills training by itself improves profits and sales more over a three-month horizon, but that neither treatment has positive or significant impacts after one year. This study faced even more difficulties than is usual with high attrition and noisy outcome measurement, so these impacts are rather 22 imprecise. For example, the 95 percent confidence interval for the impact of soft skills training on the level of monthly profits after one year is (-166%, +99%). A final area where soft skills training is being incorporated as part of entrepreneurship education is in training programs for youth. One example is the Educate! program taught in the last two years of secondary school. Preliminary four year results show this program led to lasting impacts on soft skills, but no impacts on business knowledge, earnings, or employment. 12 Over one-third of the participants were still doing tertiary education at the time of a four-year follow-up, and so it may still be too soon to measure impacts on work outcomes. However, other studies with youth also find effects diminishing over time. For example, an experiment in Tunisia which taught entrepreneurial skills during the last year of university found that positive impacts on starting new firms one-year after finishing university had disappeared over a four-year horizon (Alaref et al, 2020). The Combined Evidence on Alternatives to Traditional Training Figure 3 shows that the average impact of these alternatives to traditional training is a 14.8 percent increase in profits (95% C.I.: +7.8, +22.9). This is higher than the 10.1 percent average impact of traditional training, but the confidence intervals overlap, and the heterogeneity in impacts among studies is much larger than the differences between traditional training and its alternatives. Likewise, the average impact in Figure 4 on revenue of 11.3 percent (95% C.I.: +2.9, +19.7) is higher than the average impact of traditional training on revenue of 4.7 percent, but with the confidence intervals again overlapping. This accords with the results of several of the studies which have tested an alternative against traditional training and found somewhat larger impacts of the alternative, but often without being able to reject equality. The results suggest the promise of using some of these approaches to improve on standard training. 12 See https://www.poverty-action.org/blog/can-soft-skills-be-taught [accessed 8 June, 2020]. 23 Figure 3: Impact on Profits of Alternatives to Traditional Training Notes: Effect size is percentage change in profits. Number Trained is number of firms invited to training. Green diamond shows random effects meta-analysis estimate, and red line shows this estimate. Weight is weight random effects meta-analysis gives to study, with studies with smaller standard errors given larger weight. Where impacts were available over multiple time horizons, the longest time horizon is chosen. 24 Figure 4: Impact on Sales of Alternatives to Traditional Training Notes: Effect size is percentage change in profits. Number Trained is number of firms invited to training. Green diamond shows random effects meta-analysis estimate, and red line shows this estimate. Weight is weight random effects meta-analysis gives to study, with studies with smaller standard errors given larger weight. Where impacts were available over multiple time horizons, the longest time horizon is chosen. 25 5) Scaling up business training Classroom-based business training programs typically provide training to 15 to 40 small businesses at a time. Developing countries have hundreds of thousands, or millions of such small firms – for example, the Mexican Economic Census enumerated 4.1 million firms with 0 to 10 workers (INEGI, 2016). The big challenge for policy makers is then how to provide good quality business training services at a scale that can reach vast numbers of firms while not being prohibitively expensive for the government. I see three (potentially complimentary) approaches for achieving this goal. Approach 1: Charge firms for training and develop the market for business services A first general approach to scaling is to develop and use the market for business service provision and having firms pay at least part of the costs of training. Many governments and NGOs have offered business training for free or for a nominal fee. Charging for training can make such programs more financially sustainable, and may be a way of selecting entrepreneurs who value the training more and who will exert more effort. However, it may also prevent poorer and liquidity- constrained entrepreneurs from getting assistance. Recent work by Maffioli et al. (2020) in Jamaica explores these issues by randomly varying the price charged for training, enabling them to assess how the demand for training varies with price, and to investigate whether those who pay more for training attend training more regularly. Their business training course consisted of 40 hours divided into ten lessons of four hours each, and cost $150 to provide. They find demand falls sharply as price increases: 76 percent attend at least one class when it is offered for free, 65 percent when they have to pay a token fee of 5% ($7.50), 29 percent when charged half of the cost ($75), and only 11 percent when charged the full cost. Offering the opportunity to pay in three installments did not change willingness to pay. Poorer and smaller businesses are less likely to pay and thus attend once the price rises. However, the authors do find that those who pay higher prices attend more of the 10 sessions, suggesting a potential sunk cost or psychological effect where paying for business training makes firms value them more. Unfortunately, this study is unable to measure whether those who pay higher prices see larger business impacts from the training. A key factor to consider when charging for business training is that business training may be an experience good, which firm owners need to receive in order to assess the quality and potential 26 value to them. Suzuki et al. (2014) find suggestive evidence of this amongst knitwear producers in Vietnam, noting that the concept of receiving training to improve business performance was new to many of these firms, and that hypothetical willingness-to-pay for future training increased for those who participated in a first training. Offering the first lesson for free and/or using testimonials from other firms who have participated may be useful at helping firms learn about the value of new training services. An alternative to firm owners purchasing business training on the market is for them to use the market for business services to hire people who already have the skills that training is intended to teach. Anderson and McKenzie (2020) note that as firms grow, the entrepreneur becomes increasingly less likely to do the accounting and marketing themselves, and more likely to hire others to either do this within the firm (insourcing) or to hire external service providers (outsourcing). They test the relative effectiveness of insourcing and outsourcing compared to business training in an experiment with 753 firms with 2 to 15 employees in Nigeria. They find that using the same amount of money that the government was going to spend on business training to subsidize firms hiring expertise from the market for business services delivered greater improvements in business practices, innovation, and firm growth. Moreover, these firm owners were then more likely to go back to the market to purchase services with their own money. These market-based solutions of charging for training and facilitating the use of insourcing and outsourcing appear most applicable to firms that are above the subsistence level, have several employees, and are looking to grow. The role of the government in such cases may be to reduce information and search frictions that make it hard for firms to identify good quality providers, and potentially subsidize firms learning through experience. A vibrant marketplace of business service providers can then be one way to ensure large numbers of firms have access to training providers and business practice specialists. In contrast, these market-based solutions may be less appropriate for subsistence firms and poorer firm owners, although vouchers could still be used to help firms purchase training or other business improvement services from the market. An open question is whether such an approach could increase the quality of business training services provided, if training providers then have to compete for customers. Approach 2: Use online technologies such as edutainment, SMS, and online lessons 27 A second general approach to scaling up small business training is to use technology to dramatically lower the cost of reaching thousands or millions of people. Television offers one such medium. Two “edutainment” shows for entrepreneurship have recently been evaluated: Ruka Juu (“Jump Up”) in Tanzania (Bjorvatn et al., 2019) and El Mashroua (“The Project”) in the Arab Republic of Egypt (Barsoum et al., 2018). In both cases, the shows were reality show competitions, with weekly episodes over 11-13 weeks, that followed the journeys of young entrepreneurs as they undertook challenges teaching and testing entrepreneurial skills. Key business concepts such as market assessments, planning, advertising, record-keeping, etc. were emphasized in each episode, and both shows included both male and female contestants. Such shows definitely deliver content at scale: the Tanzanian show had 3.1 million viewers and the Egyptian one 4 million. The key question is whether they improve business knowledge. Evaluating such programs is difficult, since they are available to all who wish to watch them. Researchers use what is called a randomized encouragement design, randomly inviting and reminding one group to watch the show, and comparing the results to a group who gets invited to watch something else, or who gets no reminders. Bjorvatn et al. (2019) use this approach with a sample of 2,132 secondary school students (average age 18), and Barsoum et al. (2018) with a sample of 5,924 Egyptian youth (with an average age of 27). The findings are that these edutainment shows do seem to make viewers slightly more interested in entrepreneurship, and seeing women succeed makes viewers think it is a little easier for women to go into self- employment than they originally thought. However, neither study finds any impact on business knowledge, or on people taking actions towards starting businesses. Moreover, Bjorvatn et al. (2019) find the show appears to have discouraged students from their schooling, without leading to better business outcomes. Television is a passive medium, with no opportunity for tailoring content to the individual, or for them to receive feedback. It is therefore difficult to teach complex business practices or knowledge, but it may still play a role in increasing interest in entrepreneurship as a career. Moreover, it may be that there are impacts on a minority of viewers that cannot be picked up in surveys of a random sample of viewers: if only 0.25% of viewers start a business as a result of watching, that would still amount to 10,000 new businesses created. Two other technology-based approaches to scaling up business training are currently in the testing stage. The first is to use either SMS messages or apps on mobile phones to provide basic business 28 tips and some customized advice to small business owners. Such an approach is being used for smallholder farmers by the NGO Precision Agriculture for Development, with positive impacts on yields (Cole and Fernando, 2016). Cole et al. (2019) tested sending weekly voice messages of 3-4 minutes at a time over 21-22 weeks in experiments with 2,096 microfinance clients in the Philippines and 3,849 microfinance clients in India. These messages focus on simple rules of thumb and heuristics. They find these messages do lead to some modest improvements in business practices (0.08 to 0.13 standard deviations) over a three month follow-up, but no detectable impacts on business profits and sales, with a 95% confidence interval for the impact on profits of (-13.5%, +5.3%) in the Philippines and (-6.8%, +6.4%) in India. A recent attempt to move from general to individualized advice was conducted by Acimovic et al. (2020) in an experiment in Tanzania with 4,771 mobile money agents. This would seem an ideal setting, since by working with the mobile money operator, every transaction made by the firms could be measured, and the history of transactions used to develop daily personalized recommendations on inventory levels to hold to prevent stockouts and the need for rebalancing (e.g. “Tomorrow (Sunday), we suggest that you have 185,000 float and 125,000 cash for [mobile money operator]). Despite this rich data, they find small and statistically insignificant impacts of SMS notifications alone, and that even coupling these recommendations with in-person training first only resulted in a 2 percentage point change in stock-outs (relative to a control mean of 49 percent). An alternative is to use interactive online training programs, which can include customized and adaptive content, provide opportunities for feedback, and enable current or prospective entrepreneurs to repeat or take longer on concepts that they find more useful or challenging to understand. An ongoing randomized trial is testing the use of a 26-class training program in personal initiative and negotiations skills among over 20,000 high school students in Ecuador (Asanov and McKenzie, 2020). If successful, such an approach can deliver business training to large numbers at much lower cost than classroom-based training. Questions of how to best deliver business training online have become particularly important in many countries as a result of the COVID-19 pandemic, and the current evidence base does not have much to offer on what works in this regard. 29 Approach 3: Filter or funnel large numbers of firms to offer more support to those who will benefit the most from it Any business training program is unlikely to deliver equal benefits to all firms taking it. Some firms may benefit a lot, while others may not benefit at all, or even be hurt by trying out suggestions that turn out not to be useful for their specific firms (e.g. Karlan et al, 2015). Rather than trying to provide training to every firm, policy makers may therefore get better value for money if they can better target programs towards the subset of firms that would benefit most from it. The key challenge is figuring out how to identify such firms. Several approaches are available, but much more work is needed in this area. A first approach is to pre-screen firms to try to ensure that the firms invited to business training have aspirations and interest in firm growth, have enough complimentary resources (education and physical capital) to be able to employ the lessons learned, and are operating at a large enough size and level of sophistication for the training content. For example, Anderson et al. (2018) pre-screen South African firms on the basis of requiring firms to operate out of a physical structure, and on numeracy, education and business aspirations. Screening on size may also make it more likely the cost of training passes a cost-benefit calculation: training needs to deliver much less of a percentage increase in profits to cover $150 in training costs for a firm earning profits of $500 a month than one earning $50 a month. However, such an approach can raise concerns about equity, and it is not necessarily the case that larger, more sophisticated and inspired firm owners will benefit more from training, since they might be likely to grow anyway. A second alternative would then be to explicitly target firms on the basis of their expected treatment effects – that is, to offer training to the firms who would get the biggest gains in profits from undertaking this training. The problem is that existing studies do not offer a clear picture of what characterizes such firms. Training experiments with larger samples could be combined with new developments in machine learning to deliver this evidence in the future. With smaller subsistence-oriented firms in close-knit neighborhoods, community members may have a good sense of which businesses would benefit most from training. Hussam et al. (2020) show that among subsistence firms in dense Indian slums, peers are able to tell which firm owners will benefit most from capital grants. It would be interesting to test whether such an approach could also work for determining who would benefit most from business training. However, with larger, more growth- 30 focused firms, it can be tough for both experts and machine learning to predict which firms will grow most (McKenzie and Sansone, 2019), and at best one may be able to screen out a bottom tail of firms with the lowest chance of benefiting from training. Given the current state of knowledge on both these approaches, policy makers may prefer to use a third alternative, which is a funnel approach. The idea here is to start by providing very basic services to a large number of firms (e.g. some online course, simple benchmarking information, or a short one-hour firm visit). Then firms who demonstrate interest and undertake some improvement actions from this first engagement can be filtered into receiving a second more intermediate level of business training support, such as perhaps a first set of group training, or more detailed online courses. Knowledge tests, effort levels, and other data can then be used to observe which firms are responding more to the training, and be used to further filter the firms into a subset that receives the full package of training and support. Such an approach has the political advantage of offering some assistance to large numbers of firms, while restricting the most costly and time-consuming parts of the program to firms that demonstrate some engagement and improvement. A variant of this approach is being trialed by the Brazilian micro and small business support agency SEBRAE, although results are not yet available. 6) Conclusions and areas for additional research After the first wave of randomized experiments of business training, the lack of statistically significant impacts on profits and sales in most studies led to some pessimism about whether business training “works”. The subsequent studies in the literature have helped provide a bit more balance to this message. Coupled with thinking realistically about what effect sizes one can expect from a short training course, my reading of the literature is that i) the practices that business training tries to teach are important for businesses, but training typically does not lead to businesses adopting many of these practices; ii) a typical training program is likely to have an average impact on firm profits and sales of approximately 5 to 10 percent, which may be enough for training to pay for itself, but will be very hard for many studies to detect; and iii) the range of alternatives to traditional training do appear to offer some gains over the standard training approach alone, but we should also be realistic about how much larger these gains will be. Moreover, the optimal training program is unlikely to be one-size-fits-all, and different variants of training will be useful for different types of firms. 31 As policy makers and academics continue to test out new ideas for business training, I see three key areas that are priorities for research going forward. The first is researching different approaches to scaling up business training services, so that a good quality product can be offered to large numbers of firms. The previous section lays out an agenda for doing this. Second, I have largely discussed the average impacts for training. But different training programs are likely to benefit different types of firms, and so rather than asking “does training work?” we need to ask “what types of training work best for which types of firms?” The impacts of training are likely to be much larger if we can successfully target them. We need much larger sample sizes and better precision in measuring impacts in order to be able to develop this knowledge for targeting. Finally, although it is 18 years since Karlan and Valdivia’s (2011) first training experiment got underway, and over a decade since several other training experiments occurred, most of the training experiments only track impacts over one or two years. Longer-term follow-ups that trace impacts of training on firm survival and growth over horizons of five to ten years would be very informative in determining whether training delivers lasting improvements to entrepreneurs. A final point to note is the importance of continued research on this topic. After the first wave of experiments of business training, some journal editors and academics have expressed a view that we now have a good evidence base on business training, and that “yet another business training paper” is unlikely to be of general interest. But this review illustrates that collectively we have learned a lot through the continued accumulation of evidence, and that there remain many important open questions that need addressing. Moreover, the number of studies of business education is still tiny compared to the vast literature on general education. I therefore hope we will continue to see more studies on this topic. 32 References Acimovic, Jason, Chris Parker, David Drake and Karthik Balasubramanian (2020) “Show or Tell? Improving Inventory Support for Agent-Based Businesses at the Base of the Pyramid” Available at SSRN: https://ssrn.com/abstract=3186575 Alaref, Jumana, Stefanie Brodmann and Patrick Premand (2020) “The medium-term impact of entrepreneurship education on labor market outcomes: Experimental evidence from university graduates in Tunisia”, Labour Economics 62: 101787. Alibhai, Salman, Niklas Buehren, Michael Frese, Markus Goldstein, Sreelakshmi Papineni and Kathrin Wolf (2019) “Full Esteem Ahead? Mindset-oriented business training in Ethiopia”, World Bank Policy Research Working Paper no. 8892. Anderson, Stephen, Rajesh Chandy and Bilal Zia (2018) “Pathways to Profits: Identifying Separate Channels of Firm Growth Through Business Training”, Management Science, 64(12): 5559–5583 Anderson, Stephen, Pradeep Chintagunta, Frank Germann and Naufel Vilcassim (2020) “The Impact of Volunteer Marketers on Product Differentiation and Firm Growth: A Field Experiment with Ugandan Entrepreneurs”, Chicago Booth Paper No. 20-13 Anderson, Stephen, Christy Lazicky and Bilal Zia (2019) “Measuring the Unmeasured: Combining Technology and Behavioral Insights to Improve Measurement of Business Outcomes”, World Bank Policy Research Working Paper No. 8836. Anderson, Stephen and David McKenzie (2020) “Improving Business Practices and the Boundary of the Entrepreneur: A Randomized Experiment Comparing Training, Consulting, Insourcing and Outsourcing”, Mimeo. Arráiz, Irani, Syon Bhanot and Carla Calero (2019) “Less is More: Experimental Evidence on Heuristic-Based Business Training in Ecuador”, IDB Invest Working Paper TN No.18. Asanov, Igor and David McKenzie (2020) "Showing Life Opportunities: Increasing opportunity- driven entrepreneurship and STEM careers through online courses in schools.." AEA RCT Registry. April 20. https://doi.org/10.1257/rct.3593-3.2. Barsoum, Ghada, Bruno Crépon, Drew Gardiner, Bastien Michel, and William Parienté (2018) “Evaluating the impact of entrepreneurship edutainment in Egypt: An experimental approach with peer effects”, Mimeo. PSE. Beaman, Lori, Jeremy Magruder and Jonathan Robinson (2014) “Minding small change among small firms in Kenya”, Journal of Development Economics 108: 69-86. Berge, Lars, Kjetil Bjorvatn and Bertil Tungodden (2015) “Human and Financial Capital for Microenterprise Development: Evidence from a Field and Lab Experiment”, Management Science 61(4): 707-22. 33 Blattman, Christopher and Laura Ralston (2017) “Generating employment in poor and fragile states: Evidence from labor market and entrepreneurship programs”, Mimeo. University of Chicago. Bloom, Nicholas, Benn Eifert, Aprajit Mahajan, David McKenzie and John Roberts (2013) “Does Management Matter? Evidence from India”, Quarterly Journal of Economics, 128(1): 1-51 Bloom, Nicholas and John van Reenen (2010) “Why do management practices differ across firms and countries?”, Journal of Economic Perspectives 24(1): 203-224. Bjorvatn, Kjetil, Alexander Cappelen, Linda Helgesson Sekei, Erik Sørensen, and Bertil Tungodden (2019) “Teaching Through Television: Experimental Evidence on Entrepreneurship Education in Tanzania”, Management Science, forthcoming. Brooks, Wyatt, Kevin Donovan, and Terrence Johnson (2018) “Mentors or teachers? Microenterprise training in Kenya”, American Economic Journal: Applied Economics 10(4): 196- 221. Bruhn, Miriam, Dean Karlan, and Antoinette Schoar (2018) “The Impact of Consulting Services on Small and Medium Enterprises: Evidence from a Randomized Trial in Mexico", Journal of Political Economy, 126(2): 635-87. Bruhn, Miriam, and Bilal Zia (2013) “Stimulating Managerial Capital in Emerging Markets: The Impact of Business Training for Young Entrepreneurs.” Journal of Development Effectiveness 5(2): 232–266 Bulte, Erwin, Robert Lensink, Rosine van Velzen and Nhung Thi Hong Vu (2016) “Do Gender and Business Trainings Affect Business Outcomes? Experimental Evidence from Vietnam”, Management Science, 63(9): 2885-2902 Buvinic, Mayra, Hillary C. Johnson, Elizaveta Perova, and Firman Witoelar (2020) “Can Boosting Savings and Skills Support Female Business Owners in Indonesia? Evidence from A Randomized Controlled Trial.” CGD Working Paper 529. Washington, DC Cai, Jing, and Adam Szeidl (2018) “Interfirm Relationships and Business Performance.” Quarterly Journal of Economics, 133(3): 1229-1282. Calderon, Gabriela, Jesse Cunha, and Giacomo de Giorgi (2020) “Business Literacy and Development: Evidence from a Randomized Trial in Rural Mexico.” Economic Development and Cultural Change, 68(2): 507-540. Campos, Francisco, Michael Frese, Markus Goldstein, Leonardo Iacovone, Hillary Johnson, David McKenzie, and Mona Mensmann (2018) “Is personal initiative training a substitute or complement to the existing human capital of women? Results from a randomized trial in Togo”, AEA Papers and Proceedings 108: 256-61. Campos, Francisco, Michael Frese, Markus Goldstein, Leonardo Iacovone, Hillary Johnson, David McKenzie, and Mona Mensmann (2017) “Teaching personal initiative beats traditional training in boosting small business in West Africa”, Science, 357(6357): 1287-90 34 Chong, Alberto and Irene Velez (2020) “Business training for women entrepreneurs in the Kyrgyz Republic: evidence from a randomized controlled trial”, Journal of Development Effectiveness DOI: 10.1080/19439342.2020.1758750 Ciani, A., M. Hyland, N. Karalashvili, J. L. Keller, A. Ragoussis, T. T. Tran (2020). Making It Big: Why Developing Countries Need More Large Firms, Washington DC: World Bank Group, forthcoming Cole, Shawn and Nilesh Fernando (2016) “’Mobile’izing Agricultural Advice: Technology Adoption, Diffusion and Sustainability”, Harvard Business School Working paper no. 13-047. Cole, Shawn, Mukta Joshi and Antoinette Schoar (2019) “Heuristics on Call: The impact of mobile-phone based financial management advice”, Mimeo. IDEAS 42. Cusolito, Ana, Ernest Dautovic and David McKenzie “Can Government Intervention make firms more investment-ready? A randomized experiment in the Western Balkans”, Review of Economics and Statistics, forthcoming. Dalton, Patricio, Julius Rüschenpöhler, Burak Uras and Bilal Zia (2020) “Curating Local Knowledge: Experimental Evidence from Small Retailers in Indonesia”, Mimeo. World Bank. Dammert, Ana and Aisha Nansamba (2019) “Skills Training and Business Outcomes: Experimental Evidence from Liberia”, Partnership for Economic Policy Working paper no. 2019- 24. De Mel, Suresh, David McKenzie and Christopher Woodruff (2014) “Business training and female enterprise start-up, growth, and dynamics: Experimental evidence from Sri Lanka”, Journal of Development Economics 106: 199-210. De Mel, Suresh, David McKenzie and Christopher Woodruff (2009) “Measuring Microenterprise Profits: Must We Ask How the Sausage Is Made?”, Journal of Development Economics, 88(1): 19-31 De Mel, Suresh, David McKenzie and Christopher Woodruff (2008) “Returns to Capital in Microenterprises: Evidence from a Field Experiment”, Quarterly Journal of Economics, 123(4): 1329-72 Dimitriadis, Stefan and Rembrand Koning (2020) “Social Skills Improve Business Performance: Evidence from a Randomized Control Trial with Entrepreneurs in Togo”, Mimeo. HBS. Drexler, Alejandro, Gregory Fischer, and Antoinette Schoar (2014) “Keeping it Simple: Financial Literacy and Rules of Thumb.” American Economic Journal: Applied Economics 6(2): 1-31. Fafchamps, Marcel, David McKenzie, Simon Quinn and Christopher Woodruff (2012) “Using PDA consistency checks to increase the precision of profits and sales measurement in panels”, Journal of Development Economics 98(1): 51-57. Fafchamps, Marcel, and Simon Quinn (2016) “Networks and Manufacturing Firms in Africa: Results from a Randomized Field Experiment” World Bank Economic Review, 32(3): 656- 675. 35 Fairlie, Robert, Dean Karlan and Jonathan Zinman (2015) “Behind the GATE Experiment: Evidence on Effects of and Rationales for Subsidized Entrepreneurship Training”, American Economic Journal: Economic Policy, 7(2): 125-61. Field, Erica, Seema Jayachandran, Rohini Pande and Natalia Rigol (2016) “Friendship at work: Can peer effects catalyze female entrepreneurship?”, American Economic Journal: Economic Policy 8(2): 125-53. Fox, Louise and Alun Thomas (2016) “Africa’s got work to do: A diagnostic of youth employment challenges in Sub-Saharan Africa”, Journal of African Economies 25: AERC supplement 1: i16- i36. Giné, Xavier and Ghazala Mansuri (2020) “Money or Management? A Field Experiment on Constraints to Entrepreneurship in Rural Pakistan”, Economic Development and Cultural Change, forthcoming. Glaub, Matthias, Michael Frese, Sebastian Fischer, and Maria Hoppe (2014) “Increasing Personal Initiative in Small Business Managers or Owners Leads to Entrepreneurial Success: A Theory- Based Controlled Randomized Field Intervention for Evidence-Based Management” Academy of Management Learning and Education 13(3): 354–379. Gonzalez-Uribe, Juanita, and Michael Leatherbee (2018) “The Effects of Business Accelerators on Venture Performance: Evidence from Start-up Chile”, Review of Financial Studies 31(4): 1566- 1603. Higuchi, Yuki, Edwin Paul Mhede and Tetsushi Sonobe (2019) “Short- and Medium-Run Impacts of Management Training: an Experiment in Tanzania”, World Development 114(C): 220-36. Higuchi, Yuki, Vu Hoang Nam and Tetsushi Sonobe (2017) “Management practices, product upgrading, and enterprise survival: Evidence from randomized experiments and repeated surveys in Vietnam”, Mimeo. Nagoya City University Higuchi, Yuki, Vu Hoang Nam and Tetsushi Sonobe (2015) “Sustained impacts of Kaizen training”, Journal of Economic Behavior and Organization, 120(C): 189-206. Hussam, Reshmaan, Natalia Rigol, and Benjamin Roth (2020) “Targeting High Ability Entrepreneurs Using Community Information: Mechanism Design in the Field”, Harvard Business School Working Paper, No. 20-082 Iacovone, Leonardo, William Maloney and David McKenzie (2020) “Improving Management with Individual and Group-Based Consulting: Results from a Randomized Experiment in Colombia”, Mimeo. World Bank. INEGI (2016) Las empresas en México: Censos Económicos 2014, Instituto Nacional de Estadística y Geografía, Mexico. Karlan, Dean, Ryan Knight, and Christopher Udry (2015) “Consulting and Capital Experiments with Microenterprise Tailors in Ghana”, Journal of Economic Behavior and Organization 118: 281-302. 36 Lafortune, Jeanne, Julio Riutort, and José Tessada (2018) “Role models or individual consulting: The impact of personalizing micro-entrepreneurship training”, American Economic Journal: Applied Economics, 10(4): 222-45 Maffoli, Alessandro, David McKenzie and Diego Ubfal (2020) “Estimating the demand for business training: Evidence from Jamaica”, Mimeo. Majurin, Eva (2014) Start and improve your business (SIYB): implementation guide, International Labour Office. Geneva: ILO. Mano, Yukichi, Alhassan Iddrisu, Yutaka Yoshino, and Tetsushi Sonobe (2012) “How Can Micro and Small Enterprises in Sub- Saharan Africa Become More Productive? The Impacts of Experimental Basic Managerial Training.” World Development 40 (3): 458–68 McKenzie, David and Anna Luisa Paffhausen (2019) “Small Firm Death in Developing Countries”, Review of Economics and Statistics 101(4): 645-57. McKenzie, David and Susana Puerto (2020) “Growing Markets through Business Training for Female Entrepreneurs: A Market-Level Randomized Experiment in Kenya”, American Economic Journal: Applied Economics, forthcoming. McKenzie, David and Dario Sansone (2019) “Predicting Entrepreneurial Success is Hard: Evidence from a Business Plan Competition in Nigeria”, Journal of Development Economics, 141: article 102369. McKenzie, David and Christopher Woodruff (2017) “Business Practices in Small Firms in Developing Countries”, Management Science, 63(9): 2967-81. McKenzie, David and Christopher Woodruff (2014) “What are we learning from business training evaluations around the developing world?”, World Bank Research Observer, 29(1): 48-82 Ozimek, Adam (2013) “There is nothing magical about small businesses”, Forbes, January 8 https://www.forbes.com/sites/modeledbehavior/2013/01/08/there-is-nothing-magical-about- small-businesses/#5accbe3b7a09 [accessed 29 May 2020] Peet, Evan, Günther Fink and Wafaie Fawzi (2015) “Returns to education in developing countries: Evidence from living standards and measurement surveys”, Economics of Education Review 49: 69-90. Suzuki, Aya, Hoang Nam Vu and Tetsushi Sonobe (2014) “Willingness to pay for managerial training: A case from the knitwear industry in Northern Vietnam”, Journal of Comparative Economics 42: 693-707. Ubfal, Diego, Irani Arráiz, Diether Beuermann, Michael Frese, Alessandro Maffioli and Daniel Verch (2019) “The Impact of Soft-Skills Training for Entrepreneurs in Jamaica”, IZA Discussion Paper no. 12325. Valdivia, Martin (2015) “Business training plus for female entrepreneurship? Short and medium term experimental evidence from Peru”, Journal of Development Economics 113:33-51 37 Van Lieshout, Susanne and Pranati Mehtha (2017) The Next 15 Million: Start and Improve Your Business Global Tracer Study 2011-2015. International Labour Organization: Geneva. 38