Climate Technology Program | In Brief No. 9 Measuring Progress in Climate Innovation: Lessons from the Monitoring & Evaluation System of the Climate Technology Program Innovation in technology, business models and associated financial mechanisms holds tremendous potential to address climate change in fields such as clean energy, water and agriculture. However, measuring the progress of efforts to drive climate innovation remains a challenge. Innovation is inherently risky, has long timelines and involves numerous disparate players, all of which makes it difficult to apply traditional monitoring and evaluation (M&E) approaches. This In Brief looks at the M&E system of the World Bank Group’s Climate Technology Program (CTP) and draws lessons on how to measure progress and impact of climate innovation efforts in developing countries. Background and Context The CIC Monitoring and The World Bank Group’s Climate Technology Program (CTP) has established a global network of Climate Innovation Evaluation Journey Centers (CICs) in eight countries and regions around the The CIC M&E system evolved over time as the program grew world: Brazil, the Caribbean, Ethiopia, Ghana, Kenya, Morocco, and learnings were incorporated. Figure 1 summarizes the South Africa, and Vietnam. These centers offer seed financing, M&E journey. technical training, network linkages, and policy interventions to support clean tech ventures to launch and scale their 1. Problem Identification businesses. The M&E journey started with the question: What is the overall CICs aim to support innovation of climate-friendly and problem CICs are intended to address? The CICs aim to nurture commercial solutions that can be implemented in developing clean tech start-ups and early stage companies to tackle countries. Success for CTP would entail: (1) sustainable clean climate issues. The M&E system was designed to measure the tech intermediaries, (2) innovative and growing businesses, CICs’ effectiveness in achieving this goal and, in turn, to inform and (3) positive climate impact. The M&E system was designed their operational and strategic decisions. to cover these three programmatic goals. Figure 1. The Six Steps of the CIC M&E Journey Problem Identification: CICs Indicator Selection and Data Collection: data are aim to nurture clean tech Definition: 32 indicators collected for 200+ variables start-ups and early stage were seleceted to measure for donor reporting and CICs’ companies to tackle climate CICs’ outputs, outcomes and own analysis. issues. impact. 1 2 3 4 5 6 Logic Model: CIC services Target Setting: targets were Analysis and Learning: CICs, enable firms to grow, which esablished for each of the incubatees and investors creates climate and social indicators. can test a wide range of impact. hypotheses from the data collected. In Brief No. 9 | Page 2 2. Logic Model represent the second level of results, which relate to the goals of an intervention but usually depend on outside With a clear objective in mind, the CTP and the CICs consulted factors and require changes of behavior. For example, CIC with various ecosystem stakeholders, including donors outcomes include the business performance of supported and incubator managers, and developed a logic model to clean tech firms. Impact refers the long-lasting consequences conceptualize how the interventions will create change and of an intervention, such as CO 2 emissions avoided. Figure 2 lead to intended outcomes while accounting for assumptions illustrates how the output, outcome and impact indicators are and risks. Figure 2 presents CICs’ logic model: clean tech linked to the logic model. entrepreneurs receive needed and relevant services and support from the CIC, including training and grant financing, 4. Target Setting which enables them to grow their businesses and in turn have The CICs established targets for each of the indicators, taking a positive climate impact. into account project duration, planned activities, and post- 3. Indicator Selection and Definition incubation follow-ups. CICs were encouraged to set ambitious goals for the output indicators, which are directly linked to Based on the logic model, CTP defined and selected 32 their incubation services, but to be more conservative when quantitative indicators to measure CICs’ outputs, outcomes setting outcome indicators, since various factors that affect and impact. In M&E terminology, outputs are the immediate achievement of the targets are outside of the CICs’ control. results of an intervention and, in this case, include the number Targets are set on a yearly basis, and CICs report their results of clean tech entrepreneurs supported by CICs. Outcomes Figure 2. The Logic Model of the CICs Climate mitigation and adaptation Impact indicators: • CO 2 emissions avoided • No. of people with improved Households reached • access to modern energy/ with more climate clean water, etc. efficient products Entrepreneurship Outcome indicators: and firms grow • No. of businesses that raised $$$ Financial gap early stage finance • No. of direct jobs created, etc. addressed Enablers created and Output indicators: services provided • No. of businesses supported Consultations with • No. of of training services ecosystems players delivered In Brief No. 9 | Page 3 for each indicator on a quarterly basis to allow for regular 6. Analysis and Learning monitoring of progress and preemptive action in case activities The data collected by the M&E system will enable CICs, fall below target. investors, CIC-supported businesses, and other stakeholders 5. Data Collection to correlate and test a wide range of hypotheses. For example, CIC managers could analyze external funding raised by their Results reported for all indicators are captured through a portfolio companies to assess whether the incubation support standardized data collection process consisting of nine online is preparing a pipeline of investment-ready businesses. The forms used at different touch points between CICs, their CTP M&E team is developing a data dashboard available to partner service providers, and their businesses. This process all CIC management, with the goal of cultivating a learning captures more than 200 variables, including both information organizational culture where project leaders convert insights for donor reporting and additional information for different to action through operational decisions backed by data. Figure stakeholders to conduct analysis for their own unique needs 4 presents several illustrative examples of insights from data (see figure 3 ). analysis useful for different stakeholders. Figure 3. Data Collection Process of the CIC M&E System BASELINE ONGOING* FORM 2a. CIC PROFILE FORM 2b. CIC PROFILE FORM 7.BUSINESS FORM 8. CIC UPDATE Profile and basic Profile of CIC staff UPDATE Any changes in CIC basic information of the CIC Quarterly updates related info, knowledge products to firm level indicators, produced, knowledge product development and sharing events and media FORM 4. APPLICATION additional financials raised appearances STATUS Outcome for FORM 5. SERVICE applications received PROVISION Businesses CIC Information on Businesses service provision FORM 1. APPLICATION FORM 6. sessions to businesses When businesses/ EVALUATION OF entrepreneurs apply SERVICES for admission Feedback on service provision FORM 3. SERVICE sessions PROVIDER PROFILE Profile of CIC’s partner service provides (ex. mentors, coaches, etc.) Partner Service Providers * Baseline forms, form 1,2,3, and 4, are collected once per business or CIC. Forms 5 and 6 are collected whenever a service is provided to CIC businesses, and forms 7 and 8 are collected quarterly. In Brief No. 9 | Page 4 Figure 4. Data Analysis for XCIC Management and Potential Investors* A. For XCIC Business Stage Current Stage Management Progression Ideation R&D Testing Commercialization XCIC Total Entry Stage Businesses Ideation 100 1% (1) 1% (1) 1% (1) 3% (3) INSIGHT 2 : XCIC has ushered 32% of its businesses through R&D various stages of development 1% (1) 2% (2) 9% (9) from entry to commercialization. Testing INSIGHT 1 : 57% INSIGHT 3 : a client 4% (4) 20% (20) of firms already at regressed from commercialization at commercialization entry, so XCIC may Commercialization to testing, and this want to revise its warrants further selection process to investigation by 1% (1) 57% (57) target earlier-stage XCIC. start-ups. B. For Potential Investors and XCIC Management No. of Profitability & Haven’t rasied Raised Quandrant 1 (Q1): profitable businesses that XCIC Firms Funding funding funding have not been able to secure funding. Reported Quandrant 2 (Q2): profitable businesses that Q2 Profits & Q1 have secured funding. Funding: Profitable 10% Quandrant 3 (Q3): unprofitable businesses that 20% (10) 50 (5) were able to secure funding. Quandrant 4 (Q4): businesses that have neither turned profits nor secured funding. Q3 Q4 INSIGHT : potential investors are unlikely to prioritize firms in Q4 when they source deals Unprofitable 20% (10) 50% (25) from XCIC. XCIC’s immediate priority should be supporting these firms to transit to Q1 or Q3 as a first step for long-term success. * XCIC is an illustrative example and does not refer to any of the seven CICs. The insights are derived from the data reported by the CICs, but the numbers are revised to protect the privacy of the CICs. In Brief No. 9 | Page 5 Lessons Learned with the start-up side: they do not inherently understand that true success is rare. For example, even in a robust The implementation and ongoing evolution of the CIC M&E enabling environment, an early-stage investor might screen system provides the following lessons for other climate 100 applications but make only one or two investments. innovation projects in developing countries. Entrepreneurs working in nascent clean tech sectors in developing countries are likely to have even lower success rates, Business Growth versus Climate Impact: Which to and most clean tech start-ups will not survive the “valley of Prioritize and How to Balance? death” 1 financing gap. Climate innovation stakeholders should expect some failures, and recognize that a higher tolerance CICs represent a novel approach to addressing climate change, for risks will be compensated for with other social benefits, with a heavy focus on long-term innovation. The theory of including contribution to the local innovation system. The M&E change underpinning this approach involves first incubating system should set appropriate targets and use data analysis to early-stage clean tech ventures that have strong potential to help identify firms most likely to succeed. For example, in figure create significant climate impact as they scale up, with the 4 , the four quadrants of the chart on “profitability and funding” caveat that not all will succeed. When assessing the success could be further broken down by sector, revenue bracket, of climate innovation programs such as the CICs, there is an funding size, and many other indicators. This data analysis can inherent tension between measuring incubatees’ business help the CICs to understand which firm characteristics are most growth versus the positive climate impact they create. Progress likely to attract investors and, in turn, to identify firms with in both areas is critical, but the indicators used to assess higher probabilities of success. progress do not directly overlap. The CIC M&E experience suggests that an M&E program must balance a focus on Manage Expectations from the Outset business performance with a focus on climate impact. It offers the following lessons for achieving this balance: Because of the challenges discussed above, stakeholders need to manage their expectations of climate innovation projects at Have Patience for Climate Impact the beginning. They should not expect to measure results in the same way they would if they supported, for example, a solar farm Climate impact created by clean tech firms depends on the project which has proven technologies, known business models number of green products sold to and used by consumers. and established regulations. When evaluating a project’s value However, it takes years for even the most successful clean for money, success on business innovation must be carefully tech startups to scale and achieve sufficient sales to create weighed alongside climate impact, with both sets of results meaningful climate impact. Climate innovation projects, included in any evaluation. including the CICs, need to understand that creating climate impact takes time and must be considered a long-term Climate Innovation Requires Flexible M&E goal. Nevertheless, clean tech incubators, especially those in emerging markets, can claim short-term success if their Approaches supported businesses are showing progress in some ways, such All donor-funded incubation programs require a well-designed as turning an idea into a prototype, registering with the local M&E system to report results to the stakeholders involved. intellectual property office, starting to generate revenue, or Because of the many complexities associated with measuring raising private investments. It is important for the M&E team climate innovation, there is no one-size-fits-all M&E approach: to carefully select various indicators to measure both business any system or framework has to adapt to the specific program performance and climate impact. requirements and partner/donor expectations. Based on the CIC M&E experience, key challenges and possible solutions include Expect Some Failures the following. Climate innovation brings together the climate community and the start-up world. The climate side is usually unfamiliar 1. The “valley of death” refers to the financing gap for early stage firms between the ceiling of their initial funding and the floors of most investors. In Brief No. 9 | Page 6 Measuring the Climate Impact of Early Stage Businesses Is system needs to reflect in its design and implementation the complexity of measuring impact, which requires an Challenging understanding of clean tech ventures’ multiple benefits and When clean tech businesses are at a very early stage, what the potential trade-offs among them. is the best way to measure their climate impact? The CICs calculate the impact based on a key performance index, units Adopt a Carrot-and-Stick Approach to Data Collection sold, and assumptions of the average climate impact per Although this is not specific to clean tech incubators, difficulty clean tech product. The CTP M&E team collaborated with the in data collection adds additional complexity to the M&E World Bank’s Climate Change team by sending them detailed work and is one of the main reasons why climate innovation descriptions of the products of CIC-supported businesses. needs strong M&E capacities. The more detailed data an The Climate Change team then provided the M&E team with incubator collects, the more analysis it can conduct to inform assumptions on the average per-unit climate impact across its operational model. However, collecting firm-level data a range of products in the same product category, such as is always challenging, especially for incubators where firms cook stoves or solar panels. The CICs multiply the units sold are not charged to receive incubation services. The CTP M&E of an incubatee by the impact assumption for a specific systems collects a vast amount of data on more than 200 product category to calculate the total climate impact for that variables and has adopted a carrot-and-stick approach to data company. collection: There are a few caveats associated with this method. • Carrot : the CTP M&E team conducted a survey with The method assumes that households will replace older CIC-supported businesses to ask what they would like to products, such as a diesel pump, with clean tech products, receive in return for providing data to CICs. Based on the such as a solar pump. This might not hold true, and a full- firms’ responses, the team is experimenting with several fledged investigation is needed to confirm whether units incentives, including the provision of: sold are accompanied by changes in end users’ behavior. In addition, different stakeholders might have different impact 1. a web-based tracking system that creates a business assumptions for clean tech products. There is not yet a profile and progress using the data submitted; universally adopted set of standardized assumptions widely 2. business performance documentations that could be used in the climate community. used to get funding from a bank or other investors; and 3. performance comparisons with other businesses Be Aware of the Potential Trade-Offs between the Multiple • Stick : all CICs are advised to sign a binding agreement Benefits Created by Clean Tech Ventures with admitted firms at the beginning. The agreement Clean tech ventures not only create climate impact, they also requires firms to provide baseline and ongoing business offer other potential social benefits, such as job creation. information to the CICs. This sets entrepreneurs’ However, as the CTP learns more about CIC portfolio expectations from the start about the information they will companies, it has become clear that there are potential be asked to provide as CIC clients. trade-offs between some of the benefits created by these These incentives ensure that the valuable information provided businesses. Some businesses increased the amount of CO 2 by entrepreneurs is fed back to them in some manner and can emissions mitigated as their sales went up, but they did not be used for their own purposes, not only for meeting reporting create many direct jobs in the process. Early-stage businesses requirements. usually have a small balance sheet. As a profit-maximizing decision, they will not spend much money on hiring employees and prefer to invest limited resources in machines or technologies: Decisions are made to grow business, not to maximize social impact. A solid climate innovation M&E In Brief No. 9 | Page 7 Climate Technology Program In Brief About Us The Climate Technology Program (CTP) In Brief series is a publication of the World Bank Group’s Finance, Competitiveness, & Innovation Global Practice. CTP focuses on the growing opportunities of the clean technology sector in developing countries. Through a global network of Climate Innovation Centers, the program provides local entrepreneurs with the knowledge and resources they need to launch and scale their innovative business solutions to climate change. CTP In Brief is a series of knowledge briefs highlighting important aspects of the CTP global and in-country operations and research. Acknowledgements This brief was prepared by Xue Zheng with contributions from Yehia Eldozdar and Sarah Craig. © 2018 The World Bank Group 1818 H Street NW Washington, DC 20433