76388 Finance & PSD Impact JANUARY 2013 The Lessons from DECFP Impact Evaluations ISSUE 21 Our latest note illustrates a case where attempting to do impact evaluations can provide valuable knowledge, even if the impact evaluations ultimately are unable to be completed. Lessons for Matching Grant Programs from Failed Attempts to Evaluate Them Francisco Campos, Aidan Coville, Ana Fernandes, Markus Goldstein and David McKenzie A typical matching grant consists while two stemmed from a direct of a partial subsidy - most commonly engagement with the government. covering 50 percent of the cost - provided by In theory, matching grants satisfy a a government program to a private sector number of conditions that make firm to help finance the costs of activities to randomization a possibility: i) they involve promote exports, innovation, technological selection of individual firms; ii) the numbers upgrading, the use of business development of firms involved can be large enough to services, and, more broadly, firm growth. potentially generate enough statistical power Matching grant programs are one of for measuring impacts, and iii) data on key the most common policy tools used by outcomes may be measured reasonably well developing country governments to actively through firm surveys. facilitate micro, small, and medium Given that the government is enterprise competitiveness, and have been effectively giving away free money to firms, included in more than 60 World Bank one might expect significant demand for this projects totaling over US$1.2 billion, funding, resulting in the need for projects to funding over 100,000 micro, small and select which firms receive it. Since we medium enterprises. believe there is substantial uncertainty over Yet despite all the resources spent on which firms would best benefit from these projects, there is currently very little receiving these funds, our suggestion was credible evidence as to whether or not these for randomized evaluation based on an grants spur firms to undertake innovative oversubscription design. The idea here activities that they otherwise would not have would be to make the matching grant done, or merely subsidize firms for actions programs open for all firms meeting certain they would take anyway. basic eligibility criteria, and then randomly Since firms self-select into whether select which firms would be awarded the they apply for such programs, and then the grants from among eligible applicants. programs decide which applicants receive funding, attempts to compare outcomes for What happened? matching grant recipients to non-recipients Out of the seven projects that we are likely to be biased. discussed impact evaluations with, five initially agreed to implement projects with Attempted Experiments an oversubscription-based randomization We set out to design randomized experiment included, while the other two experiments to prospectively evaluate seven had encouragement designs planned. matching grant programs in six African However, in practice, we were countries. Five were to be supported through unable to implement any of the randomized World Bank loans and technical support, experiments successfully. The main reasons were: Do you have a project you want evaluated? DECRG-FP researchers are always looking for opportunities to work with colleagues in the Bank and IFC. If you would like to ask our experts for advice or to collaborate on an evaluation, contact us care of the Impact editor, David McKenzie (dmckenzie@worldbank.org) 1. Lack of applications: despite giving away were current on taxes; requiring firms to free subsidies, programs struggled to get go through procurement processes like enough applicants, resulting in insufficient getting three bids in writing for services, eligible applicants to randomize among. and paying upfront for services with 2. Repeated delays and changes in personnel: delayed reimbursement; and restricting implementation delays of over a year or what grants could be used for. more led to changes in government  Incentives facing project staff: project personnel, reversing some of the buy-in for staff were typically on fixed wage the evaluation; it also meant we ran up contracts, giving no incentive to try and against impact evaluation funding. get more applications. Implications Why is it so hard to give away free money? 1. Matching grant programs need to  Political economy and capture: these change the mindset from picking subsidies were viewed by some winners to picking positive governments and partners as something treatment effects: the latter is to give their constituents: so we found even harder to judge, making Chambers of Commerce lobbying to impact evaluation even more keep eligibility conditions such that only their members would be eligible, while important. local governments competing with 2. Focus more on eligibility criteria national governments in programs. and making it easy for firms to  Overly strict eligibility criteria: apply, and to get the money once requiring firms to be registered with they are awarded it. audited tax accounts tended to exclude 3. There are also several lessons for most firms in many African countries: designing impact evaluations of criteria were set on the idea of which such projects: these include using firms would grow fastest, not which methods for small samples; firms would see the greatest benefit having more realistic expections from the program. on time frames; and conducting  Last mile issues and red tape: many of “little IE” or impact evaluation the conditions imposed by the World Bank and governments made it difficult on different program design and burdensome for even eligible firms features. to apply and receive money. Examples include: requiring firms to get letters from the tax department to show they For further reading see: Francisco Campos, Aidan Coville, Ana Fernandes, Markus Goldstein and David McKenzie (2012) “Learning from the experiments that never happened: Lessons from trying to conduct randomized evaluations of matching grant programs in Africa” World Bank Policy Research Working Paper no. 6296 Recent impact notes are available on our website: http://econ.worldbank.org/programs/finance/impact