n G EF LESSONS N37t* 1 4 www.gefweb.org July 2002 "6,Lessonis L:earned ;in A/Mantfacturing and Marketing of Energy Efficient Products Introduction The GEF Operational Strategy is built around 10 operationai IN THIS , ? , ' ' k ,. principles, and one of those principles addresses monitoring In 2001, GEF, 'oniro ring and etaluation (M&E,jtami and evaluation: "The GEF will ensure that its programs and i'aliaiexI'the use fmarkettran'sformation approaches projects are monitored and evaluated on a regular basis." to siimulaietlhe manufacture andis efergy5efcient~ This Lessons Note shares practical lessons from the monitor- prohict i ';Dight "proijects were.-`revieweda; 'seven, O ing and evaluation (M&E) experience of four GEF projects, w iich 'are :being implemnentedSin Chima, (3)WM, Alexico', so that other projects may benefit from these experiences. (w)hPoland(I) and,TPail; m .1 2 wh/le,thu eighth, a multicountry initiative,, is b'ing carried out by''seven The projects discussed in this Lessons Notes are: countries on four conti'ents.: iit'eaminatio'fof pI)roject, imipacts, sustaintability, and',,replication,, "the * Poland Efficient Lighting Project (PELP) review, generated some key lesso,I s/i'niili it lI''.' anl , effectiv project,monritoring and cialita,to pl(7,1 cal * Thailand Promotion of Electricity Energy Efficiency '' I elpiOit ilu Je ilmarket "Ii h;fo'ii,za,aini One (Thai DSM) "'lof the most crucial lessons is the iniponatiic e, i. A16 E, a. an' integral 'part of project design and nimp.hle'int'iiditll. * Mexico High Efficiency Lighting Pilot (Ilumex) Tlhis',Lesssons Notes' ilustrates 'these lessons"b'ase'd ,on' , .fur ofthe'reviewed projects.- >, ,'. '.................f * Efficient Lighting Initiative (ELI) f of th ie pjects GEF Lessons Notes are' written toi ssist project These projects were included in a review of the GEF's climate designers, riu,iui.i ;, .and.' t'laifan'r ii 'EF&'' change projects that target the manufacture and marketing of inplementing and 'executing agencies.and beyond W energy-efficient products. A review of this cluster of projects 'welcome your feedback and 'u5e' siinIu ii'iiC 'i was carried out in 2001 as part of the overall assessment of ongoing practical.vatifthe Notes series the performance of GEF-supported activities in the climate ' ' change focal area.' s Monitoring and Evaluation Integral to All GEF Projects Jeirle Hari ud Findings from the careful monitoring and evaluation of Sepno, .oturii w ad Ei nui,',a C' 'ordi lluor project indicators have a variety of uses. First and foremost, when properly integrated into a project's design and imple- mentation, M&E can provide feedback to project manag- ers, enabling them to modify implementation in response to changed circumstances; in this way, M&E becomes part of any project's risk mitigation strategy (see ELI example in Iy Box 1). M&E results also have value for other interested par-F LE C ties, such as national govevaments or elenric utilities, and 'The complete review is "The GEF Energy-Efficient Product Portfolio: Emerging Experience and Lessons" by Sabrina Birner and Eric Martinot (Washington, DC: Global Environment Facility, 2002). The Poland Efficient Lighting Project (PELP), financed by the GEF swering questions on lighting use and purchase habits and mailing and implemented through the International Finance Corpora- in the card, consumers were entered into a drawing for prizes. tion (IFC), set out to reduce Polish greenhouse gas emissions by These cards were wildly popular. In each lighting season, PELP increasing the use of energy-efficient compact fluorescent lamps received over 10,000 cards-so many cards, in fact, that the post (CFLs) At the time the project was designed, CFLs were expen- office delivered them daily by the bagfull The cards became a sive and not well-known, and consumer demand for them was database of purchasers of PELP-labeled CFLs, the market research low. PELP sought to address this combination of high prices and firm was then able to contact purchasers 1 to 2 years later and low demand on two fronts First, PELP lowered CFL prices with a learn whether their CFL was still being used. subsidy to qualified manufacturers, and, second, it increased con- sumer awareness of the lamps through a mass-media campaign An American consulting firm designed the PELP evaluation, and PELP's CFL promotions took place in the autumn and winter of a Polish market research firm carried out the associated market 1995/96 and 1996/97 PELP also hoped to have a lasting impact on research; neither firm had any stake in the project's outcome The the market beyond the end of its promotions. use of independent firms helped strengthen the credibility of the evaluation PELP's design included an M&E plan that would assess the proj- ect's effectiveness. The M&E plan aimed to. The PELP evaluation did run into some difficulties. The evalua- tion plan depended on obtaining sales data from manufacturers, * Quantify direct program impacts What were the kWh and car- but manufacturers were reluctant to share this very sensitive data. bon dioxide emissions savings from the sale and installation of Also, at the beginning of the project, certain nuances of consumer CFLs labeled and subsidized through PELP? questionnaires were lost in translation. This was remedied by * Quantify indirect impacts In what ways did PELP lead to long- adding a specialized translator to the evaluation team Greater term tuse of on-site meters to measure lighting energy use also would have helped resolve some data ambiguities. Finally, in retrospect, * Summarize project experience- What are the lessons from PELP a "comparison country" would have been helpful as a control that would be helpful to future GEF projects? against which to measure changes in the Polish market. PELP evaluators sought information from different market actors In spite of these difficulties, the PELP evaluation was able to to answer these questions To track consumers' awareness and use obtain credible data that showed, from several different perspec- of GFLs, a Polish market research firm interviewed both randomly tives, PELP's effectiveness in increasing sales of CFLs Some of the selected consumers and purchasers of PELP-labeled CFLs To gain evaluation's important findings include: insight into the CFL market's evolution, the firm also surveyed re- * CFL penetration increased from one in 10 households before tailers about parameters such as CFL sales and the number of dif- PELP to one in three households 1 year after the close of PELP ferent models available From manufacturers, evaluators obtained information on sales of PELP-labeled CFLs. * Prices of CFLs declined by more than 34 percent d6uring the program. These price decreases were sustained after the pro- To better track PELP's impact, data collection took place before, gram and have not reverted to pre-subsidy levels. 4 during, and after the project A baseline study2 was conducted before PELP's CFL promotion, and consumers and retailers were * CFLs became more widely stocked (more stores carry CFLs and surveyed during the program and into late 1998, over a year and a carry more models), and data from 1 year after the program in- half after PELP's completion Program managers used data collect- dicated that this retail availability was sustained. ed over the course of the project to respond to changing market conditions and adjust the implementation strategy * Ninety-seven percent of CFL purchasers intended to replace their CFLs with another CFL when the current one burns out Two evaluation tools, a media clipping service and consumer re- sponse cards, were especially valuable PELP hired a media clip- * Print media coverage increased and shifted from explaining ping service to gather articles that mentioned CFLs These articles what CFLs are to describing where and how to best use them, gave valuable insight into the general population's perception of indicating the now-common use of CFLs CFLs In 1996, CFLs were reported as a "new" phenomenon A year later, news articles described them as an "expensive but vi- These indicators demonstrate a widespread and sustainable able" option for Polish households By 1998, they were considered change in the residential market for CFLs in Poland The evalu- a "popular" form of lighting, with many benefits to the people, the ation also measured PELP's overall impact-a savings of 2,320 country, and the global environment GWh, or 2 8 million metric tons of C02 avoided, at a cost of $1 39 per ton These strong M&E results supported the creation of a fol- Participating manufacturers were required to package PELP-la- low-on GEF project, the Efficient Lighting Initiative (ELI), which is beled CFLs with consumer response cards In exchange for an- currently being implemented by the IFC in seven countries. 2 A baseline study determines the "business as usual" scenario-number of lamps sold, lamp prices, number of retailers selling the lamps, etc.-before the project begins. This information is essential for measuring the project's impact after completion. Before beginning its M&E activities, the Thai DSM project GEF Recommendations for M&E Indicators relied on engineering estimates alone for calculating the For organizations in the midst of planning an M&E strategy, program's impact on peak load reduction. At the GEF and it can be helpful to consider seven core project indicators.4 World Bank's insistence, they undertook a more rigorous They are presented below as guidance; however, the relative evaluation effort, which included dedicated studies to assess importance of an indicator depends on the project's activities. such parameters as the coincidence factor (the amount of For example, for a project seeking mainly to develop local time an appliance is on during a utility's demand peak) and energy-efficient manufacturing capacity, policy development free riders (people who would have taken action even with- (indicator five below) is less relevant. out the program), that led to greater accuracy. This increased accuracy allowed managers to better integrate the impact of 1. Energy production or savings and installed capacities DSM into their planning. 2. Technology cost trajectories 3. Business and supporting services development The M&E plan for ELI (see Box 1) creates a symbiotic rela- 4. Financing availability and mechanisms tionship between M&E and implementation. ELI's mid-term 5. Policy development process evaluation will provide data to the evaluation team as 6. Awareness and understanding of technologies well as feedback to the implementation team. 7. Energy consumption, fuel-use patterns, and impacts on end users Practical Guidance for Designing an M&E Plan Budgeting for M&E M&E should be an explicit line item in any proposal submitted to the GEE The M&E line item should include allocations for pre-implementation baseline surveys, ongoing data gathering during implementation, and a post-program review of the persis- tence of the program's impacts. Table 1: M&E Criteria for GEF Projects Work Program Inclusion Describe how the project design has incorporated lessons from similar proj- ects in the past. Describe approach for project M&E system, based on the project logical framework, including the following elements: * Specify indicators for objectives and outputs, including intermediate benchmarks and means of measurement * Outline organizational arrangement for implementing M&E * Indicate total cost of M&E (may be reflected in total project cost). Project Brief Review or CEO Endorsement Finalize M&E plan, including: * Detailed budget * Final organizational arrangements for implementing M&E * Indicators for project activities, including intermediate benchmarks and means of measurement. Implementation/Completion On an annual basis during project implementation, submit project implementa- tion report to GEF M&E as input into the Project Implementation Report Prepare Project Completion Report and submit it to the GEF M&E department. Source, GEF Project Cycle, October, 2000, Document GEF/C 16/INF7 i Source: "Measuring Results from Climate Change Programs: Performance Indicators for GEE" Monitoring and Evaluation Working Paper 4, September 2000. BoxI1: Overview of the Monitoring'and Evaluation Plan of,the Efficient Lighting,lnitiative (ELI) 'VVyhe Efficient Lighting,lnitiative (ELI) aims to reducegreenhouse 'gas emissions bytiniresin'g the p r,'.r'r of,ene, i Ir,,- . ''technologies in, seven countries Argentina, Czech Republic, Hungary Latvia, Per6 PhIbpr-" ard'South/Afrca ELIhas incorporated a rigorous M&E component iito its workplan to address such questions as Can.a coordinated set of b'est practi&e'agproaches to market intervention be ' t .el, ' mplenl;nd n, oi .r,e of,'countries,,,$ 'and conditions? -' ' , S . Can such a program be successful at'altering energy technology supply and purchasing behaviors in a significant poirtion of', the targeted'popuiations` ', . , , , .' What'impacts can these changes in market conditions and behaviors be expected to have,on gree.nhru"e g'asemissions re . ductions in the h,ear terr and into the future? ' * - * 'Ilf successful, can this type of worldwide program activity be repicad for, te cholocs,ttand othe'r gre'en ," h'ouse gas m itigation strategies'. , , , " >4,, ''I '-4 ,,:-s 7o' ,: i",'9$ ';Z' M&E Activities 'G6ilded by such'questions, three major M&E activities wili be carried c,ut in each country, parallel to the project's imple ,mentabion,. x ' ' '"' ''" The, Process Evaluation will examine the design, delivery, and, follow-up :1 ,-e e,-s a- .,--;d .,ki program,, mplements3tlao prbvides'two occasions for feedback to project implementers ,during the baseline market i resent ar,d 'r.rrnq th-' mdI.Eerni p r., e .. ;evaluation Inevitably, exogenous political, economic or technological changes'will affec- . -ien--rr0iraon i'. fe-dt Lv buida gr.p-' portuity foir analysis arid modification Into program Implementation Itrrs hoped that the leedback b-r.-.-r. r 1SE 3n' ,nr.n-r,r,rr, will help create among ELI implementers a culture that emphasizes flexibility and'respocnsibi i I- Dr r -,.Ir- rather ri-,r9 Tn. :r dh"r-nh.,-r 10 The Impact,Evaluation will attempt to quantify ELI's energy and environmental impacts It,will try to controlf. r e r rIIuer. such as changes in economic conditions so that conclusions can be drawn about;the specific impa6cts of,the EL I it ir measure.both" ,direct and indirect impacts on energy consumption 'and then estimate greenhouse gas err;- `d tI d, r i n: in electricity - -generation , , -. ,,* , / / The Market;Transformation"Analysis will examine the program's impact on theuknowledge'.~ ,..ir', .r.J itud."rjes ,-t rnsk.- P ':(producers,,suppiers an'd end-Lsers) and assess the extent'to which changes 'ri besustan'ed Th,s lc- ghse oi ihe e.aluat,crlee:t ' the fact that the'ELl is a market'trangformation program To track impacts bejynd t'4e`s`6rtfterm -narker 3tin[,uIj thar orcur4 dw.rs. implehientation, M&E data,collecti6n and analysis will extend several years` ,'n /Ythe proram'send. ''' 'Training ELI also seeks to ircrease understanding amorng prograr imp!ementers 6f the 'value,'ofi'arefuJlly monitoring market'interven-,', 'tion strategies Only by incorporating M&E practices into thei p'rograms cari stakeholherscet.irniwhet4 r 4 rrrrworked ars intercded, information on project accomplishments builds support f6r'future' efforts 'Independence The ELI M&E'team consists of (1) North' Amehcan energy and Cra.,rnnme,,r4I proyr,n eJaltiat,c.r. leading the.' 'technical'design and analysis,'(2) an internatiorial market research firm,with affiliat- . e *n '-r rEs;,i3 .-" r.'bY ir t r. ollectio'n"" '"and (3) energy,fficiency/environmental experts'within the seven 'countries; serving as liaisons§ for'providing" inr-co.,ut, . e . ni ete. ,.prptatin ~To keeo.the mon~itonng and,evaluation'free from'co4nficts of interest'the M&E r.-m nrr. hr e .-,o n,, n ci errt r, rh.e ELI im hplenmentation' and icIo stake in its outcome , , ' " ' ' ' ' '' Cand' Fi' n`iCHg`mar- ' ' ' " ''/S '-.,Candor Finally,the evaluation' does nrot seek to'assess individuals performance but rarher a'strr.'3g' -ei1ec* ' ,s;ke{' Therefore,'the evaluation'seeks 'candor i'n all contacts'to enable.buldinggan,)accurate pictu1re on howmthe proprl3nr h3- ceer, .mpIC. Imented- t is only with full knowledge and understan'ding of.'onditions, barriers actions'andIoutcomes tha't arccurate e5 mer,rs car. b. 4 obtained,;and conclusions can reached about ELI's effectiveness.' ' ' Source Adapted from "IFC/GEF Efficient Lighting initiative (ELI) Monitoring and Evaluation Overview,` which can be accessed at www e'ficientig9htgn9 ret Box 2: The Strong Cost of a Weak Evaluation 'lumex P '..,., ' 'S The Mexico High Efficiency-Lighting Pilot'(1ume twast irt f is ki to te tund d byt GEF T ,,u the prorect rh.,e rex,car, utI ity CFE leased energy-efficient compact fluorescent lamps (CFLs) to its customers Thece lar-,p:s cost more rh ar. re.juttar lIaht bu,lbs buw ose gne-quarter less energy. -Though he prj,.1-t elf appeared to ac hireve a ,cJ reuIts. many qu-atr r"ere lt ,jnanr .erp.d ard cp- 'portunities forVprogram improvement were lost tbe,-u .e ike procrr3mJ ' &.-.lJ,Iu.n - superficial and -sis dela,ed until after tF,e -lose of the program. : 'The impleinenntation plan for llumex included a mid r-Trr, rn- .e that vi.uld prc-c,1 ,,;, d o a ppiotunity, ' ne-tun, irole-ct rniplerrenia. 'tion: In advance of the mid-term review, the CFE (the ,r, nn mpIrr,enr;r,,-rg thi" program) was ^h,re ciri -ultarn; to r,on,ro. CFL u5ge and-assess their i'pact on energy consumptionr. But'CFE did ,t hr,re rhe onsultants in a timely manner 9r,d rhe VV.:rld B.nk %*.h,;:h *s supervising the prqject, was not forceful enough in,demanding.that' CFE proceed with the rn,1d ern, e0.5Igur.c*n a; 3 re-Sult an, lind,ngs of recommeridations from the mid-termn review mission were&largely irrelevant.for project rr.plrre-lr,rarron and rh-e prolect 1._.;r 3r. anpEr. tunity for yaluable feedback that could have helpe`d mproeA its design and 'rnpl-rr-nienEat or Furtherniore, "the evaluation that did take place didrnot; -e ,h,, ;d;. a.r tE nre-aure'u-,h ba-,. p3rarrer-:rs ai cc- eiect,.er,et; en.,- rorimental impact: financial performance,,and erer g, usa'ge., .. ,, / The World Bankis mplernentationCcrnpletr ecr, ,ep.or rir hmn I :th r;ll ofitskey lessons eird Th.e denr,hr,o-eest appropriate project moinitoring Indicators and rhe ,-7.tular o,llect.Or .Di data .} 1he h nr:p.-me,er.tr.rq agencycire'esiernt,al te, ir, e.aluar.rn ongoing pilot.proiect perfo-m'nce.,Without ,glcrou; m.r.iornq r.n q rc, ectIs urceis orffaiiure will be difficult ro delern-mine Source Implementation Completion Report, Mexico Htgh Efficiency Lighting Project, World Bank, December 1998. Resources GEF Project Cycle'(1995)- http //www.gefweb.org/public/procycle htm Monitoring and Evaluation of Market Development in World Bank GEF Climate Change Projects, World Bank (1998) This report can be downloaded from' http //www gefweb org/Resultsandlmpact/Monitoring-Evaluation/M-E.Procedures/m_e_procedures html Martinot, Eric, and Sabrina Birner. The GEF Energy-Efficient Product Portfolio. Emerging Experience and Lessons. This report can be downloaded from http //gefweb.org/Resultsandlmpact/Monitoring-Evaluation/Evaluationstudies/evaluationstudies html Terms of Reference for the Evaluation of the Efficient Lighting Initiative (ELI) http://www efficientlighting org/html/documents/evaluation/evaluation.html Workplan for the Evaluation of the Efficient Lighting Initative (ELI)' www.efficientlighting org http://www.efficientlighting,org/html/documents/evaluatoon/evaluation.html Evaluation of the IFC/GEF Poland Efficient Lighting Project CFL Subsidy Program Final Report Edition 1. http //www.ifc org/enviro/EPU/EEfficiency/PELP/pelp htm General information on ELI M&E: http.//wwwefficientlighting org/m-and-e/index html Other GEF Monitoring and Evaluation Publications of Interest Feedback and Suggestions The GEF Energy-Efficient Product Portfolio: Emerging Experience We hope the GEF Lessons Notes series will be a catalyst and Lessons is available on the GEF website (www.gefweb.org) for an ongoing dialogue on what is working, what is not, or from the GEF Monitoring and Evaluation team. Earlier issues and how people involved in the GEF have found solu- of GEF Lessons Notes can also be obtained from the website or by wrtn. t s tions to challenges that face all of us. We welcome your reactions to this issue. We would also like your sugges- If you would like to be on the mailing list for future issues of tions of topics of interest to you Please send an e-mail to GEF Lessons Notes, please contact us at the references below. geflessons@gefweb.org or contact us at the coordinates Please let us know whether you wish to receive an electronic listed below. version or a hard copy, and which language (English, French, or Spanish) you would prefer. ' r5 .' GEF Secretariat'Monitoring and Evaluationl,.Program"r ;1 818 H Street, NW,, ,,,,, ;>., Washin'gton, ;DC 20433, U AS. ,",'' :Telephohn: (202),458-2548 e' Faxt.(202)"522-3240 - s. ma'il: geflessons@qefweb org / . '''',A ..www.gefweb.org can be an asset to a project's implementing organization, as was 2. M&E should be integrated into a project's design to take the case with the Thai DSM program evaluation (implemented advantages of opportunities for data gathering. The consum- by the Electricity Generating Authority of Thailand (EGAT), an er response cards are an excellent example of how a project's electric utility). Many GEF projects use innovative approaches fundamental activity (for example, selling lamps) can be whose documentation provides guidance to future initiatives (see designed to generate valuable evaluation data (for example, PELP case study). Last but not the least, M&E can inform the enclosing a consumer response card with each lamp sold). GEF about a project's impacts and highlight valuable experiences If M&E is tacked on after a project has been fully designed, for a range of GEF stakeholders. such opportunities may be lost. For all these reasons, the GEF requires that M&E be fully The ELI evaluation offers another example of the advantages integrated into a project's development and implementation. of integrating M&E into a project's design. The data elements Table 1 specifies GEF's requirements for M&E at each stage that the ELI M&E team identified for collection in market of the project cycle. This Lessons Note focuses on the Poland assessment surveys served both as baseline data for M&E and Efficient Lighting Project (PELP) M&E plan, draws some key market data that allowed the implementation team to refine lessons, and further illustrates those lessons with examples from the program design. other projects. It condudes with some practical guidance for designing an M&E plan. 3. Rigorous M&E that is undertaken by an independent, professional organization strengthens a project's credibility. Lessons Learned from the Case Study An evaluation undertaken directly by the program imple- menters loses credibility. Therefore, the evaluation should be The PELP evaluation case study sheds light on the characteris- conducted by an independent and professional third party, tics of a good evaluation plan: paying careful attention to promoting good communication between the implementation and evaluation teams. Having 1. Monitoring and evaluation should take place before, during, a single organization undertake all market research ensures and after a project. consistency of studies over time. If a local firm is implement- ing an M&E study designed by a foreign firm, questionnaires * Before the project starts, a baseline study should be con- should be translated locally with special care. ducted to establish the "business as usual" parameters that the project is trying to change. Without a baseline, it is im- The Thai DSM project, which was implemented by the Elec- possible to determine a project's impact.3 And while there can tricity Generating Authority of Thailand (EGAT), combined be tension between the implementer's urge to start program in-house evaluation with independent verification. EGAT activities and the delay associated with a baseline study, expe- conducted its own DSM program evaluation, which was rience shows that the 3-6 months spent conducting a baseline then reviewed by an independent monitoring and evaluation study before the project begins is time well spent. agent. While this approach provided M&E independently * During implementation, the parameters surveyed in the verified results, it also led to some tensions around questions baseline study should be monitored, to yield feedback for of cultural sensitivity. program managers. A mid-term evaluation may also assess the Rigorous M&E makes the data on a program's impact more process a project uses (seminars, subsidies, advertising, etc.), credible both within the implementing organization and helping to determine whether modifications could make it vis-a-vis other institutions. M&E gave EGAT a high level of confidence in the results and helped their program gain in- * After implementation, research should be conducted to assess ternational recognition. the persistence of the project's impacts over time. This is espe- cially important for market transformation programs, as they 4. M&E can provide valuable feedback to managers. M&E seek to have a long-term impact. should not be a burden on project managers, nor should it be a "project police force" that creates an uncomfortable This "before, during, and after" approach is illustrated in the atmosphere. Rather, the M&E plan should be designed as a evaluation plan for the Efficient Lighting Initiative (ELI), as tool that helps managers monitor a project's evolution and presented in Box 1. The Mexican Ilumex project provides a empowers them to make mid-course corrections, if necessary. counterexample of the difficulties encountered when evaluation A close relationship between M&E and implementation can is delayed until after a project's end (see Box 2). help the program be more responsive to change. I Evaluation specialists distinguish between impact evaluation, whose goal is to determine the impact of a project and whose result is usually expressed in figures (GWh savings, $/ton of avoided pollutants, etc.), and process evaluation, which assesses the effectiveness of the means used by the project to achieve the goal.