Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GT Enhancing MSME Productivity Project (P112011) Report Number : ICRR0020712 1. Project Data Project ID Project Name P112011 GT Enhancing MSME Productivity Project Country Practice Area(Lead) Guatemala Trade & Competitiveness L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-80000 31-Dec-2017 32,000,000.00 Bank Approval Date Closing Date (Actual) 03-Mar-2011 30-Jun-2016 IBRD/IDA (USD) Grants (USD) Original Commitment 32,000,000.00 0.00 Revised Commitment 4,502,382.84 0.00 Actual 4,502,382.84 0.00 Prepared by Reviewed by ICR Review Coordinator Group Antonio M. Ollero Ridley Nelson Christopher David Nelson IEGFP (Unit 3) 2. Project Objectives and Components a. Objectives According to the Loan Agreement of November 2012 (page 5), the project development objective (PDO) of the Guatemala Enhancing Micro, Small and Medium Enterprise Productivity Project was: "to stimulate the growth of micro, small and medium enterprises (MSMEs) in selected value chains". The Amendment to the Loan Agreement of July 2014 and the Second Amendment to the Loan Agreement of February 2016 retained the PDO. Page 1 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GT Enhancing MSME Productivity Project (P112011) b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? Yes d. Components Original Project The original project had three components (Project Appraisal Document, pages 5-10). Improving and Promoting Business Development Services (US$9.7 estimate at appraisal, US$2.18 million actual) aimed to strengthen the capacity of the Ministry of Economy (MINECO) to lead and coordinate the development efforts of the public private sectors in support of MSMEs, including improving business development services and quality services and conducting pilots to develop new products to increase access to business development services and financial services. The component had four sub- components: (a) promoting a culture of quality and improving quality services; (b) strengthening business development services; (c) supporting pilots for new product development; and, (d) strengthening MINECO’s Vice Ministry for MSMEs. Creating Productive Value Chains (US$19.0 million estimate at appraisal, US$1.16 million actual) aimed to improve the competitiveness of MSMEs through: (a) the provision of technical assistance and training to facilitate the design of value chain sub-project proposals; and, (b) the provision of grants to partially finance selected value chain sub-projects, in the tourism and agribusiness sectors. According to the PAD (pages 9- 10), the tourism value chain activities that could be supported include: the development of new tourism packages with high MSME participation; upgrades of heritage assets; creation of safe zones for targeted tourism routes; training of tourist guides; strengthening of tourism organizations; introduction of clean production technologies in small hotels; creation of a joint web site; and, familiarization trips by international tour operators. The agribusiness value chain activities that could be supported include: the development of new products; market intelligence research; cold chains and other small infrastructure improvements; niche marketing schemes; and, product certifications and seals. Project Management and Monitoring (US$2.52 million estimate at appraisal, US$1.19 million actual) aimed: to support the MINECO Project Implementation Unit (PIU) with technical assistance, training, equipment, and operational funds to carry out the implementation management and monitoring and evaluation of the project; and, to provide technical assistance for conduct of the financial audit of the project. First Revised Project With the first restructuring of July 2014, the first revised project retained the three components with their original cost allocations, but changed some aspects of the second component as follows (Amendment to Page 2 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GT Enhancing MSME Productivity Project (P112011) the Loan Agreement of July 2014, pages 1-3): Creating Productive Value Chains (US$19.0 million estimate at appraisal, US$1.16 million actual) aimed to improve the competitiveness and productivity of MSMEs through: (a) the provision of technical assistance and training; and, (b) the provision of grants to partially finance selected value chain sub- projects, in the tourism and agribusiness sectors. Two changes were made with this statement: improving the productivity of MSMEs was added as an intermediate objective; and, the scope of technical assistance and training was expanded to include activities other than the design of value chain sub-project proposals. Moreover, certain implementation arrangements for this component were revised: (a) the Steering Committee was replaced by an Advisory Committee; (b) the Bank’s role in selection of value chain sub-projects was changed from granting approval to interposing no objection to the decision of which value chains were to be supported; and, (c) project beneficiaries were redefined to be members of a value chain and/or MSMEs or private companies rather than value chain working groups. Second Revised Project With the second restructuring in February 2016, the second revised project was scaled down from a US$32.0 million to a US$7.0 million operation and the scope of the project components was reduced to that which could be completed by end-April 2016, two months before a new project closing date of end- June 2016. The three components would become (Restructuring Paper, pages 7-9): Improving and Promoting Business Development Services (US$2.58 estimate at the second restructuring, or 27 percent of the original cost; US$2.18 million actual) aimed to strengthen the capacity of the Ministry of Economy (MINECO) to lead and coordinate the development efforts of the public private sectors in support of MSMEs, including improving business development services and quality services and conducting pilots to develop new products to increase access to business development services and financial services. According to the Restructuring Paper (page 8), the cancellation of component activities would significantly reduce the impact of the component with regard to: (a) the strengthening of the National Quality System; (b) the promotion of quality standards among MSMEs; and, (c) the provision of business development services to MSMEs. Creating Productive Value Chains (US$2.95 million estimate at the second restructuring, or 16 percent of the original cost; US$1.16 million actual) aimed to improve the competitiveness of MSMEs through: (a) the provision of technical assistance and training to facilitate the design of value chain sub-project proposals; and, (b) the provision of grants to partially finance selected value chain sub-projects, in the tourism and agribusiness sectors. According to the Restructuring paper (page 8), the reduction of the scope of the component would mean that: (a) only 1-2 sub-projects of the original 3-4 planned for each of the six selected value chains would have been chosen for implementation; (b) each of the sub-projects would have a reduced scope; and (c) fewer MSMEs would be supported by the component activities, which would also be fewer in number. Project Management and Monitoring (US$1.39 million estimate at the second restructuring, or 55 Page 3 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GT Enhancing MSME Productivity Project (P112011) percent of the original cost; US$1.19 million actual) aimed: to support the MINECO Project Implementation Unit (PIU) with technical assistance, training, equipment, and operational funds to carry out the implementation management and monitoring and evaluation of the project; and, to provide technical assistance for conduct of the financial audit of the project. Activities permissible in May-June 2016 would be those related to closing the project, including making final payments to providers, preparing financial reports, and conducting audits. Project Revisions and Split Evaluation Although the two restructurings of the project did not modify the PDO of the operation, they significantly changed the scale and scope of the project and revised the results indicators by which the achievement of the PDO were to be measured. For this reason, this ICR Review conducts a split evaluation of the project, one for the original design of the project and one each for the two project revisions. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: The Project Appraisal Document (PAD) estimated the original project cost, including an unallocated amount and front-end fees, at US$32.0 million. The second restructuring of the project reduced the project cost to US$6.92 million, following the cancellation of some activities under the first component and the scaling down of the second component. Financing: The Bank financed US$32.0 million of the original project cost. With the second restructuring of the project, the Bank, jointly with MINECO, cancelled US$25.0 million of the original loan and revised the funding commitment to US$7.0 million. The amount disbursed at project closing was US$4.61 million. Borrower Contribution: The Loan Agreement and its two amendments did not require the Republic of Guatemala to provide any counterpart funding to the project. Dates: The project was approved in March 3, 2011 and became effective in December 18, 2012. It was restructured twice, the first, in July 11, 2014, with 30.4 percent of the US$4.61 million project funds disbursed; and the second, in February 3, 2016, with another 66.8 percent of the project funds disbursed. The project closed in June 30, 2016, a year and a half ahead of the original closing date of December 31, 2017. 3. Relevance of Objectives & Design a. Relevance of Objectives Original Project The objective of the original project continues to be relevant to economic conditions and development Page 4 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GT Enhancing MSME Productivity Project (P112011) priorities in Guatemala at project closing. MSMEs employ 75 percent of the country’s economically active population and are the large providers of employment in the rural and indigenous areas where poverty, which is comparatively high in Guatemala at 47 percent, remains especially persistent. A previous value chain analysis identified significant barriers to the growth of MSMEs: (a) the need for a culture of innovation and quality; (b) the poor availability of the national quality system; (c) low levels of technology transfer; (d) few productive relationships between the private sector and the academic community; and, most crucially, (e) the lack of integration of MSMEs into the production, processing, and marketing networks in the economy. Consequently, concerted efforts to enhance the productivity of MSMEs, to facilitate their access to financing, and to link them to new markets and technologies would strengthen their participation in the national and global economy. The enhanced market activity and productivity of MSMEs would yield economic gains, which would in turn help reduce poverty, considering the MSMEs critical role in the labor market. Similarly, the objective of the original project remains well aligned at project closing with the strategic objectives of the World Bank Group’s Country Partnership Framework (CPF) for Guatemala for FY2017-20. The CPF focuses on fostering the inclusion of vulnerable groups as the first pillar of the Bank Group’s assistance, and on addressing bottlenecks to sustainable growth in the country as the second pillar. The objective to stimulate the growth of MSMEs in selected value chains is well aligned with both pillars of the CPF. First Revised Project As the restructuring of July 2014 did not change the PDO of the project, the objective of first revised project remains relevant to current economic conditions and development priorities in Guatemala and aligned with the strategy of the WBG CPF for Guatemala for FY2017-20 for the same reasons cited for the original project. Second Revised Project Similarly, the objective of second revised project remains relevant to current economic conditions and development priorities in Guatemala and aligned with the strategy of the WBG CPF for Guatemala for FY2017-20 for the same reasons cited for the original project. Rating Revised Rating Substantial Substantial b. Relevance of Design Original Project Page 5 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GT Enhancing MSME Productivity Project (P112011) The causal chain from components to outcomes and to the PDO as articulated in the original Results Framework appears convincing. Improving the capacities and capabilities of public agencies and private entities that provide business development services and quality services to MSMEs strengthens the support system for MSMEs in the country. Directly supporting MSMEs by improving their linkages to other actors in their value chains strengthens the competitiveness of MSMEs. The outcomes --- more competitive MSMEs and an improved support system for MSMEs --- enables the growth of MSMEs, broadly defined, thus achieving the PDO. Nonetheless, there were a couple of deficiencies with some elements of project design, including with some elements of the Results Framework. The PDO, while kept simple, lacked precision --- "growth" of MSMEs could be, and was in the M&E design, interpreted in varied ways (see Section 7.a). The value chains to be supported by the project in pursuit of enhancing the growth of SMSEs were left undefined, with the prescribed process for eventually selecting them during project implementation being, in addition, poorly planned. The sub-component "Supporting pilots for new product development" was inaccurately labeled. Most of the reforms under this sub-component were directed at promoting the growth of microfinance institutions (MFIs) -- - creating a registry for non-profit MFIs, automating the administration of the MSME Development Trust Fund including by developing an information technology platform to upload reports of MFIs, and supporting a feasibility study to initiate work on a Credit Guarantee Facility. The PAD and the ICR do not discuss the choice of the specific investment loan (SIL) as the financing instrument for the project. But, considering that the project combines sub-grants to private value chain sub- projects and technical assistance and training to public and private business services providers, the SIL was a suitable choice. First Revised Project The restructuring of July 2014 improved the design of the project (see Section 2.d). "Improving the productivity of MSMEs" was added as an intermediate objective, resulting in a more robust statement of the expected outcome of the project --- "More competitive and productive MSMEs and an improved support system for MSMEs". In addition, the scope of technical assistance and training activities in support of value chains was expanded, strengthening the set of project activities aimed at attaining the project outcomes and the PDO. The restructuring also improved the set of results indicators listed in the original Results Framework and the original M&E Design of the project (see Section 10.a). Outcome indicators that were difficult to quantify were replaced with more practical measures. Output indicators that did not even have targets were dropped. Targets for the remaining indicators were reset to more realistic levels. And, new output indicators that would better reflect the expected project results were added. Second Revised Project The restructuring of February 2016 greatly reduced the size and scope of the activities to what had been achieved in the four years of project implementation since end-2012. The implication of cancelling several Page 6 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GT Enhancing MSME Productivity Project (P112011) business development service activities was that the impact of the project with regard to providing business development services to MSMEs, promoting quality standards among MSMEs, and strengthening the National Quality System would be reduced (see Section 2.d). Similarly, the implication of limiting the number of value chain sub-projects was that fewer MSMEs would benefit from value chain support activities than originally targeted (see Section 2.d). These decisions practically diluted the effectiveness of the project as an MSME competitiveness and productivity support operation. Rating Revised Rating Modest Modest 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective To stimulate the growth of micro, small and medium enterprises (MSMEs) in selected value chains. Rationale Outputs • Improving and Promoting Business Development Services: No calibration services were provided to MSMEs using the mobile laboratory funded by the project. The National Metrology Center (CENAME) announced the availability of the mobile laboratory in July 2016, but the laboratory had yet to be deployed by the project closing date. The original target was for the CENAME to provide 230 calibration services to MSMEs by 2017 (this output indicator was adjusted at the first restructuring, but dropped at the second restructuring). • No certifications were provided to MSMEs. The original target was to provide 15 certifications to MSMEs by 2017 (this output indicator was adjusted at the first restructuring, but dropped at the second restructuring). • No hits were made (a hit is a request to a web servers for a file) on the online platform that would provide information on international standards and a database of certified companies. The Ministry of Economy (MINECO) had only started to analyze how technology platforms offered the planned information services. The original target was for the online platform to record 90,000 hits by 2017 (this output indicator was dropped at the first restructuring). • No hits were made on the online directory that would provide a database of business development service providers. The Ministry of Economy (MINECO) had only started to analyze how technology platforms offered the planned information services. The original target was for the online platform to record Page 7 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GT Enhancing MSME Productivity Project (P112011) 180,000 hits by 2017 (this output indicator was dropped at the first restructuring). • Creating Productive Value Chains: No workers were trained in the tourism value chain, as the tourism value chain was not selected to be supported under the project. No original target was set for this output indicator (this output indicator was dropped at the first restructuring). • No workers were trained in the e-tourism platform, as the tourism value chain was not selected to be supported under the project. No original was target set for this output indicator (this output indicator was dropped at the first restructuring). • No tourists used tourism packages supported by the project, as the tourism value chain was not selected to be supported under the project. No original target was set for this outcome indicator (this output indicator was dropped at the first restructuring). • No companies in any agribusiness value chain were considered compliant with Sanitary and Phytosanitary Standards (SPS) (i.e. that agricultural goods crossing output borders be sanitary with regards to pests and pathogens). No original target set for this output indicator (this output indicator was dropped at the first restructuring). • No determination was made of the price gap between the U.S. import prices of produce of Guatemala compared to that of other Latin American countries. The expected output --- a narrowing price gap --- could not have been attributable to the project (this output indicator was dropped at the first restructuring). • There was no data to determine the overall number of producers participating in training and outreach programs supported by the project. The only data available was that 355 members of eight producer organizations in the potato value chain received training on good agricultural and manufacturing practices and 217 potato seed growers received training on potato seed production. No original target was set for this output indicator (this output indicator was dropped at the first restructuring). Outcomes The growth of MSMEs in selected value chains was not stimulated, as originally planned. • No MSME participated in the value chain working groups, as no value chain working group was established. The Advisory Committee selected six value chains to receive support from MINECO. The project activities went only as far as creating a value chain committee of 12-32 members for each value chain, with the committee preparing a strategic vision, an action plan, and a list of producer organization sub-projects for the value chain (this PDO-level outcome indicator was dropped at the first restructuring). • There was no data to determine the increase in the value-added in the value chains that could be attributed to the project. Project support in terms of technical assistance, including training, and investment funding to implement the value chain action plans was limited to the potato value chain, with 1,209 members. Investment in sub-projects reportedly made a 20 percent rate of return, although no evidence is presented in the ICR to support this claim. The original target was to attain a 20 percent increase in the value-added in the value chains supported by the project by 2017 (this PDO-level outcome indicator was dropped at the first restructuring). Page 8 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GT Enhancing MSME Productivity Project (P112011) Rating Negligible PHREVDELTBL PHINNERREVISEDTBL Objective 1 Revision 1 Revised Objective To stimulate the growth of micro, small and medium enterprises (MSMEs) in selected value chains. Revised Rationale Outputs • Improving and Promoting Business Development Services: No calibration and testing services were provided to MSMEs using the mobile laboratory. The revised target was for the CENAME to provide 450 calibration and testing services to MSMEs by 2017 (this output indicator, revised at the first restructuring, was dropped at the second restructuring). • No normalization, certification, accreditation and metrology verification services were provided to MSMEs. The revised target was to provide 475 of these services to MSMEs by 2017 (this output indicator, revised at the first restructuring, was dropped at the second restructuring). • No business development service providers were validated, categorized, and registered in a new business development service online database. The website hosting the database went online in November 2016, but no business service provider had as yet registered with the platform. The target was for 300 business development service providers to be registered in the database by 2017 (this output indicator, added at the first restructuring, was dropped at the second restructuring). • A total 396 MSMEs received business development services provided by MINECO. The target was to have 185 MSMEs receive business development services from MINECO by 2017 (this output indicator, added at the first restructuring, was revised with a higher target at the second restructuring). • No microfinance institution was registered in the new Microfinance Registry. The Registry was launched in November 2016, after the newly enacted Microfinance Institutions Law made the registration of microfinance institutions mandatory. However, the regulations to implement the law had yet to be approved. The target was to have 80 microfinance institutions in the registry by 2017 (this output indicator, added at the first restructuring, was dropped at the second restructuring). • Creating Productive Value Chains: Zero percent of the MSMEs participating in the tourism value chain received services from the project. The tourism value chain was not selected to be supported under the project. The target was for 20 percent of MSMEs in the tourism value chain to receive services by 2017 (this output indicator, added at the first restructuring, was dropped at the second restructuring). • There was no data of the percentage of MSMEs participating in the agribusiness value chain that received services from the project. The target was for 30 percent of MSMEs in the agribusiness value chain to receive services by 2017 (this output indicator, added at the first restructuring, was dropped at the second restructuring). Page 9 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GT Enhancing MSME Productivity Project (P112011) Outcomes The growth of MSMEs in selected value chains was not stimulated, as planned in the first restructuring. • The number of MSMEs that received support from at least one activity conducted by the project was 451 (this PDO-level outcome indicator was added at the first restructuring but did not carry a target at that time). • There was no data to determine the increase in the revenues of MSMEs participating in value chains supported by the project. The target was to attain a 20 percent increase in revenues by 2017 (this PDO- level outcome indicator was added at the first restructuring, but dropped at the second restructuring). Revised Rating Negligible PHINNERREVISEDTBL Objective 1 Revision 2 Revised Objective To stimulate the growth of micro, small and medium enterprises (MSMEs) in selected value chains. Revised Rationale Outputs • Improving and Promoting Business Development Services: A total 396 MSMEs received business development services provided by MINECO. The revised target was for 370 MSMEs to receive business development services (this output indicator was revised with a higher target at the second restructuring). • Creating Productive Value Chains: Six Value Chain Strategic Plans were developed, exactly meeting the target --- the export vegetables value chain with 26 producer organizations; the potatoes value chain, with 52 producer organization and 5,915 members; the beans value chain with 25 producer organizations and 9,601 members; the cocoa value chain with 19 producer organizations, 13 pre-cooperatives and 3,912 members; the cardamom value chain with 52 producer organizations and over 10,000 members; and the papaya value chain with 18 producer organizations and 2,168 members (this output indicator was added at the second restructuring). • A total 12 Competitiveness Subproject Proposals were developed. The target was to develop 17 proposals by 2017 (this output indicator was added at the second restructuring). • Only 27 MSME producer groups, with 244 member, were formalized. The target was to formalize 30 MSME producer groups by 2017 as part of a pilot to help the producer groups and their members gain access to international markets (this output indicator was added at the second restructuring). • Project Management and Monitoring: A total 188 MINECO personnel were trained including through study tours, exceeding the target. The target was to provide training to 175 MINECO personnel (this output indicator was added at the second restructuring). Page 10 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GT Enhancing MSME Productivity Project (P112011) • A total 28 beneficiaries were trained in good manufacturing and business practices, exactly meeting the target (this output indicator was added at the second restructuring). Outcomes There was no evidence that the growth of MSMEs in selected value chains was stimulated, as planned in the second restructuring. The early closure of the project precluded the collection and analysis of any data to assess the efficacy of the support provided by the project to the potato value chain. • The number of MSMEs that received support from at least one activity implemented by the project was 451. The target was to have 400 MSMEs receive support from at least one project activity by 2017 (this PDO-level outcome indicator was added at the first restructuring but without any target; it was given a target of 400 MSMEs at the second restructuring). • As formulated, the indicator is a measure of the project’s activity (input) --- the number of MSME’s receiving project support --- rather than outcome --- the growth of MSMEs. There is no evidence that the receipt by any MSME of support from the project would have stimulated the growth of the beneficiary MSME. Revised Rating Negligible PHREVISEDTBL 5. Efficiency Lacking an economic efficiency measure, and given operational cost and time inefficiencies, the efficiency of the project is assessed as Modest. Economic Efficiency The PAD did not calculate an economic rate of return for the project. Because the value chain working groups that would determine the scope of the project activities had yet to be selected, the PAD (page 14) claimed that the project could not as yet be subject to an economic cost-benefit analysis. Rather, a technical committee would subsequently evaluate the costs and benefits of each project activity using net present value analysis to determine that the activity was economically justified. The ICR (page 17) did not attempt to quantify the benefits of the project. It reports, however, that for the potato value chain that was supported by the project, the economic rate of return of the project investments and capacity building activities would likely reach 17 percent. The rest of the project activities did not lend themselves to a formal economic or financial analysis. Page 11 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GT Enhancing MSME Productivity Project (P112011) Operational Efficiency The operational efficiency of the project exhibited significant weaknesses. Project implementation was subject to numerous delays (ICR, pages 6-7). Needing approval by the Guatemalan Congress, the project took a year and nine months from the Bank approval date in March 2011 to become effective in December 2012. There were no provisions to advance project implementation either, in the form of a project preparation advance facility or a retroactive financing scheme. Many of the vital project decisions, principally the choice of the value chains, took considerable time to be settled upon. Project implementation spanned three administrations and three Vice Ministers of the Economy, each of whom took time to become familiar with the project and each of whom had different priorities as to which value chain to support. The six value chains were selected in the first quarter of 2015, before the Mid- Term Review of April 2015. The Government had a weak implementation capacity, particularly with procurement. Conflicting Bank and Government standards and practices were not helpful. Moreover, the Guatemalan political scandals of 2015 exacerbated the problem: bidding on contracts were delayed as private contractors feared entering into transactions with the Government, less they be tainted with corruption. The ICR (page 17) reports there was at least one aspect of procurement where the project was cost-efficient. By bundling purchases into large packages, the project was able to procure, at reduced prices, higher quality products that were not typically available in the local market. The savings on procurement of spray pumps and plastic containers, for example, amounted to 20-30 percent of original estimates. The Project Implementation Unit was also responsible for implementing the Bank-supported Rural Economic Development Program (PRED). Attention to the PRED took precedence over that to the project. Only when the PRED closed in 2014, could the PIU devote its efforts to the project. Joint agreement between the Bank and MINECO meant that the project closed a year earlier than planned, but only because the project, which had been restructured twice, was deemed unlikely to complete its remaining activities. Efficiency Rating Negligible a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) Page 12 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GT Enhancing MSME Productivity Project (P112011) 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Original Project The relevance of objectives is rated as High. The relevance of design is rated as Modest. The efficacy of the project is assessed as Negligible. The efficiency of the project is assessed as Negligible. The outcome for the project is rated as Highly Unsatisfactory, as there were severe shortcomings in the operation's achievement of its objectives and in its efficiency.　 First Revised Project The relevance of objectives is rated as High. The relevance of design is rated as Substantial. The efficacy of the project is assessed as Negligible. The efficiency of the project is assessed as Negligible. The outcome for the project is rated as Highly Unsatisfactory, as there were severe shortcomings in the operation's achievement of its objectives and in its efficiency. Second Revised Project The relevance of objectives is rated as High. The relevance of design is rated as Substantial. The efficacy of the project is assessed as Negligible. The efficiency of the project is assessed as Negligible. The outcome for the project is rated as Highly Unsatisfactory, as there were severe shortcomings in the operation's achievement of its objectives and in its efficiency. Split Rating Based on a split rating, with the original project rated Highly Unsatisfactory (30.4 percent weight), the first revised project rated Highly Unsatisfactory (66.8 percent weight), and the second revised project rated Highly Unsatisfactory (2.8 percent weight), the outcome of the project is rated Highly Unsatisfactory. The project had severe shortcomings. a. Outcome Rating Highly Unsatisfactory 7. Rationale for Risk to Development Outcome Rating Page 13 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GT Enhancing MSME Productivity Project (P112011) The risk to the sustainability of the development outcome, measuring this in terms of the little that was achieved, is assessed as High. Political Risk: The risk that political commitment to the objective to stimulate the growth of MSMEs in the value chains selected under the project is rated as High. It took three administrations a considerable amount of time and effort to understand the project. Each administration had its own development philosophy and strategy. And each administration had its own choice of value chains to support. It is likely that the debate on whether it would be more optimal to target large firms or to support MSMEs will be renewed as governments change, as will the question of which value chains to prioritize. Institutional Risk: The risk that the primary institution targeted for support under this project, MINECO, might not have the capacity to lead and coordinate efforts of the public and private sectors to support MSEs is rated as High. Considering that many of the activities planned to strengthen the ability of the MINECO to provide business development services to MSMEs and to promote quality standards among MSMEs had been cancelled, it appears unlikely that MINECO will have the capacity to adequately carry out these mandates in the future. Financing Risk: The risk that financing may not be readily available to fund the implementation of action plans for the value chains is rated as High. The inability of the Government to utilize the financing made available under the project for these activities will likely weigh on the prospects for future donor financing for value chain support projects. The financing risk will likely be more pronounced for five of the six value chains that were unable to make much progress under the project --- export vegetables, beans, cocoa, cardamom, and papaya. a. Risk to Development Outcome Rating High 8. Assessment of Bank Performance a. Quality-at-Entry The Bank attempted to pioneer a novel approach to supporting value chains, with little past experience elsewhere supporting value chains in lending operations other than through matching grant schemes. Given the pilot nature of the operation, the Bank left many aspects of project design missing or incomplete. The Bank did not clearly articulate the link between the first component of the project, with it its four sub- areas, and the PDO. In particular, analytical work that would have better defined and designed the second component of the project --- the analysis of potential value chains, the design of a methodology to select value chains, and the selection of the value chains --- was not completed during project preparation but pushed back to the project implementation phase of the operation. The ICR (page 4) reports that project preparation was hastened to take advantage of the Government’s commitment to pass the project through Page 14 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GT Enhancing MSME Productivity Project (P112011) Congress before the general elections of September 2011. Consequently, design issues that would have taken more than eleven months to consider and complete were postponed to the project implementation phase. In fact, the Bank did not prepare an adequate Results Framework that would have established the causal chain from the project activities to the PDO. Moreover, the project preparation team defined PDO-level outcome indicators that were either difficult if not impossible to measure (the US$ value added in the value chains), or did not reflect an outcome but rather represented an input to (i.e., activity of) the project (the number of firms participating in value chain working groups). The Bank did not prepare an Operations Manual to guide the implementation of the project. An Operations Manual would have addressed fiduciary arrangements, including for procurement which, for a variety of reasons including conflicting Bank and Government procurement standards and practices, eventually stalled project implementation. Quality-at-Entry Rating Unsatisfactory b. Quality of supervision The Bank fielded timely missions to supervise the implementation of the project. A team member was based at the Bank country office to provide direct support to the implementing agency. Ten supervision missions over the five-year life of the operation filed an equal number of Implementation Status and Results Reports (ISRs). The missions recognized implementation problems early on, rating the project’s "Progress Toward the Achievement of the PDO" (DO) and "Overall Implementation Progress" (IP) as both Moderately Unsatisfactory by the third ISR in June 2012, six months before project effectiveness in December 2012. The Bank approved a Level II restructuring requested by the Government in July 2014. The restructuring amended the Loan Agreement, but not the project objective, to: modify the definition of beneficiaries of the project; broaden the objective of the technical assistance (TA) and training activities; replace the Steering Committee with an Advisory Committee and clarify the responsibility of the committee members; alter the Bank’s role in the selection of the value chains; and, revise the project outcome and output indicators in the M&E plan (see Section 9.b). The Bank carried out a Mid-Term Review (MTR) of the project a month earlier than originally scheduled in April 2015. The six value chains to be supported by the project had only been recently selected, in the first quarter of 2015, notwithstanding unsettled issues with the selection methodology. The Bank and the Government reached an agreement in principle on a substantial restructuring of the project moving forward, including by revising the scope of the project, cancelling half the loan, and simplifying the project fiduciary arrangements. However, the restructuring would not be pursued as envisioned in the MTR. A Level I restructuring would have needed Guatemalan Congressional approval, a process that would take 22 months to complete (ICR, page 6). The Bank, instead, approved a second Level II restructuring of the project in February 2016 that cancelled Page 15 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GT Enhancing MSME Productivity Project (P112011) US$25.0 million of the US$32.0 million original loan and advanced the project closing date by a year and a half from December 2017 to June 2016. In view of the decision to close the project early, the restructuring also revised: the scope of the components and sub-components; the cost allocations and disbursement estimates; the implementation schedule; and, the project’s M&E plan, including the outcome and output indicators (see Section 9.b), all in agreement with the Government. The ICR (page 19) reports that the Bank exerted its best effort to supervise the procurement and the financial management of the project, in the face of numerous fiduciary challenges throughout project implementation. Nonetheless, a more effective supervision of the project was also handicapped by the frequent turnover in the Bank Task Team Leader (TTL) position. The Bank appointed three TTLs in the first two and half years of project implementation after effectiveness. The leadership stabilized only after the fourth TTL was named, after which decision making became more consistent, according to the ICR. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Unsatisfactory 9. Assessment of Borrower Performance a. Government Performance The project needed Guatemalan congressional approval and took one year and nine months from Bank Board approval to become effective in December 2012 (or, one year and three months from the Guatemalan general elections of September 2011). The project spanned three administrations each of which needed time to understand the project, and each of which had its own development rural development philosophy and advanced its own development priority (i.e., support MSMEs in rural areas versus support large firms and let the economic rewards trickle down to the rest of the economy). Each Vice Minister of the Economy of the three administrations also had their own preferences of which value chain to support. According to the ICR (page 19), Government appears to have firmed up its commitment to the project by the Mid-Term Review in April 2015. The Government finally settled on the six value chains that would receive project support in the first quarter of 2015 --- two years and three months from project effectiveness. Implementation capacity remained a nagging issue throughout project implementation, however; and the Government could have given the problem greater attention. Page 16 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GT Enhancing MSME Productivity Project (P112011) Government Performance Rating Unsatisfactory b. Implementing Agency Performance The Ministry of Economy (MINECO) acted as implementing agency of the project. The same Project Implementation Unit (PIU) at MINECO, that was responsible for implementing the Bank- funded Project to Support a Rural Economic Development Program (PDER) from 2006 to 2014, was also tasked to execute the activities of the project. With the PDER activities still ongoing as the project became effective in December 2012, the PIU staff could not fully attend to the project until the PDER closed in December 2014. The PIU staff also did not have a full understanding of the project, according to the ICR (page 21). The thematic and topical overlap between the PDER and the project appears to have confounded the PIU staff. Besides, the PIU did not have budget executions powers, and could not implement the project activities effectively. More seriously, the PIU was deficient in carrying out the procurement activities. Implementing Agency Performance Rating Unsatisfactory Overall Borrower Performance Rating Unsatisfactory 10. M&E Design, Implementation, & Utilization a. M&E Design Original Project The M&E plan for the original project called for measuring the results of the operation with two PDO-level outcome indicators and ten component-level output (intermediate outcome) indicators. There were shortcomings with the two outcome indicators. The first, "value increase per unit, in the respective value chains", defined as "the increase in the amount spent per visitor" in the tourism value chain and as "the value increase per product unit" in the agribusiness value chain, would have been difficult to measure in the agribusiness value chain. Neither indicator had a baseline value, for which the targets were stated as "10 percent of the baseline" by the third year and "20 percent of the baseline" by the sixth year. The second, "the increase in the number of MSMEs participating in value chain working groups", appears to be a measure of an input to, rather than an outcome of, the project. Page 17 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GT Enhancing MSME Productivity Project (P112011) The four output indicators for the first component of the project, did not have baseline values or targets: "the number of calibration services provided to MSMEs", "the number of accreditations provided to MSMEs", "the number of hits on the online platform (Web site) providing information on online standards and listing of certified companies", and "the number of hits on the online directory of business development service providers". The six output indicators for the second component of the project did not have baseline values. Four did not have targets either: "the number of hits in the project-supported e-tourism platform", "the number of tourists using project-supported tourism packages", "the number of companies in the value chain that were compliant with sanitary and phytosanitary (SPS) standards", and "the price gap between U.S. import prices of Guatemalan produce relative to those of other Latin American countries". Only two had targets, but based on unspecified baseline values: "the number of workers trained in the tourism value chain by programs supported by the project", for which the targets were "15 percent of the baseline" by the third year and "30 percent of the baseline" by the sixth year, and "the number of producers participating in project-supported training and outreach programs", for which the targets were "15 percent of the baseline" by the third year and "30 percent of the baseline" by the sixth year. Moreover, the latter two appear to be measures of inputs to, rather than outputs from, the project component. First Revised Project Following the first restructuring of July 2014, the M&E plan for the first revised project modified the two original PDO-level outcome indicators. With "value-added" difficult to measure, the first outcome indicator became "the increase in the revenue of MSMEs in the value chains". "The number of MSMEs that received support from at least one activity implemented by the project" replaced the second original indicator. Three of the four original output indicators for the first component of the project were modified and the fourth dropped to become: "the number of calibration and testing services provided to MSMEs"; "the number of accreditations, certification, normalization, and metrology verification services provided by MSMEs"; and "the number of business development service providers registered in the new online database. Four new output indicators were added: "the number of microfinance institutions (MFIs) registered in the new MFI registry"; "the number of municipalities in which MSMEs can receive business development services provided by the MINECO MSME Directorate"; "the volume of Bank support for the institutional development of SMEs"; and, "the volume of Bank support for the enabling environment of SMEs". One of the six output indicators for the second component of the project was modified to become: "the percentage of MSMEs receiving services from the project related to the tourism value chain". The remaining five were dropped. A new indicator was added: "the percentage of project beneficiaries that can implement clean production improvements". Second Revised Project Following the second restructuring of the project in February 2016, the M&E plan for the second revised project was overhauled anew. Only one PDO-level outcome indicator remained: "the number of MSMEs that received support from at least one activity implemented by the project", an indicator that appeared to be an input measure rather than one that reflected the outcome of the project. Page 18 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GT Enhancing MSME Productivity Project (P112011) All seven output indicators for the first component of the project were dropped and replaced by two new indicators: "the number of MINECO staff trained", and "the number of MSMEs that receive business development services provided by the MINECO Business Development Services Unit". All three output indicators for the second component of the project were dropped and replaced by three new indicators: "the number of beneficiary MSMEs trained in good manufacturing and business practices"; "the number of value chain strategic plans elaborated"; and, "the number of competitiveness sub-project proposals developed under the project". b. M&E Implementation Original Project and First Revised Project M&E implementation was less a matter of reporting the results of the project --- all indicator values were still zero by the time of the first (as reported in ISR No. 6 of May 2014) and the second restructuring (ISR No. 9 of December 2015) --- than a matter of revising the outcome and output indicators, in the search for more specific, quantifiable, and attributable measures. Second Revised Project Only by the project closing date could the supervision report (ISR No. 10 of June 2016) record any non-zero value for the sole PDO-level outcome indicator, the two output indicators for the first component of the project, and the four output indicators for the second component. c. M&E Utilization Original Project and First Revised Project Data gathered in the M&E implementation of the project, essentially showing little to no progress with the project activities and outputs, helped with the decision to restructure the project in July 2014 and in February 2016. Second Revised Project The recommendation in the MTR of April 2015 to revise the scope of the project and reduce the loan amount by half was rejected in favor of an early closure of the project and a the reduction of the loan amount by about 80 percent in view of the likely lengthy procedure for obtaining a Guatemalan Congressional approval for the MTR plan (see Section 8.b). M&E Quality Rating Page 19 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GT Enhancing MSME Productivity Project (P112011) Modest 11. Other Issues a. Safeguards Environmental Safeguards: The project was classified as an Environmental Category B at appraisal and triggered Operational Policies 4.01 (Environmental Assessment), 4.04 (Natural Habitats), 4.09 (Pest Management), and 4.11 (Physical Cultural Resources). The Government prepared An Environmental and Social Management Framework (ESMF) that was disclosed in country and through the Bank InfoShop in November 2010. The ICR (page 10) reports that compliance with environmental safeguards was rated Satisfactory at project closing. MINECO obtained environmental licenses for two value chain sub-projects that managed to initiate activities before project closing. The PIU prepared Environmental Management Plans for the potato value chain and for the beans value chain. Earlier, the PIU had planned to continue work previously initiated under the PDER to prepare product environmental guides, in collaboration with the value chain working groups. Social Safeguards: The project also triggered Operational Policies 4.10 (Indigenous Peoples) and 4.12 (Involuntary Resettlement). The Government prepared an Indigenous Peoples Planning Framework (IPPF) and a Resettlement Framework, both of which were disclosed in country and through the Bank InfoShop in November-December 2010. Compliance with social safeguards was rated Satisfactory at project closing. The PIU completed social assessments that identified the stakeholders for each value chain, conducted socialization workshops, and carried out consultations with indigenous producers and with women participating in the value chains. The social assessments raised concerns about the use of child labor in post-harvest activities in the export vegetable value chain, to which the Bank responded by declaring that no disbursements be made under the second component of the project until the extent of the issue was known and until mitigation measures were instituted. The Bank subsequently lifted the disbursements bar after mitigation measures, including training in the value chains on child labor issues, were made operational. The PIU also initiated consultations preparatory to producing Indigenous Peoples Plans (IPPs) for the potato value chain and for the export vegetables value chain. As the project did not involve any involuntary land acquisition, the PIU did not prepare any Resettlement Action Plan. b. Fiduciary Compliance Procurement: Procurement was assessed as Moderately Unsatisfactory. Fiduciary performance remained weak despite the constant provision of technical support by the Bank. The Government appeared not to have fully understood and did not adhere to Bank’s Procurement Guidelines and Bank’s Consultant Guidelines as required in the Loan Agreement. Rather, the Government insisted on following national requirements and practices which were often in conflict with Bank standards and guidelines. MINECO repeatedly required of Page 20 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GT Enhancing MSME Productivity Project (P112011) bidders and contractors documents that were inapplicable or unnecessary. MINECO required all contracts be broken down into one-year phases, with a separate contract for each year, and each of which one-year contract was subject to an approval process. Where contracts spanned the jurisdiction of several ministries, MINECO mandated the approval of the contract by each Vice Minister. The result was a procurement system that: discouraged and impeded the participation of potential bidders, required the execution of multiple one- year contracts, involved a lengthy and unpredictable contract approval process, delayed the delivery of goods and services, hindered contract administration, defied transparency, and was generally cost-inefficient. Financial Management: Financial management was assessed as Moderately Satisfactory at project closing. The PIU did not have independent budget execution powers. Rather, PIU financial management activities, including contract payments, were subject to the approval of MINECO, a process that was often lengthy and inefficient and sometimes complicated and discretionary. An offer by the Bank to review budget management procedures was not taken up by the Government. The submission by MINECO of withdrawal applications were at times delayed, and so were the submission of statements of expenses. On the positive side, MINECO submitted financial reports and audited financial statements on time. An assessment of the financial management performance in February 2016 determined that the PIU complied with the project financial reporting requirements, with the information complete and acceptable and the records up to date. c. Unintended impacts (Positive or Negative) The project did not have any unintended impacts. d. Other No other issues were raised by the ICR. 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment There were severe shortcomings in the Outcome Unsatisfactory Highly Unsatisfactory achievement of objectives and in project efficiency. Risk to Development High High --- Outcome Moderately Moderately Bank Performance --- Unsatisfactory Unsatisfactory Borrower Performance Unsatisfactory Unsatisfactory --- Quality of ICR Substantial --- Page 21 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GT Enhancing MSME Productivity Project (P112011) Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons Five lessons are drawn from the ICR (pages 20-23), with some adaptation. A project that adopts the programmatic approach must consider the institutional capacity of project participants to design, develop, and implement eligible sub-projects that are aligned with the program objectives. In this case, the project postponed making the basic decision of which agribusiness value chains to support to the implementation phase of the project, expecting MSMEs and other value chain members to readily prepare action plans and eligible projects. Many aspects of project design were also left incomplete, including results indicators and targets. Considering the weak institutional capacity in the country, the project could have benefitted from substantive project preparation work, possibly supported by a technical assistance (TA) grant or loan, to identify and analyze the potential value chains, draw value chain strategy and action plans, and prepare eligible sub-projects for funding by the time of project appraisal. The project lending operation could then have concentrated on supporting the implementation of the value chain sub-projects. An Operations Manual can serve as a useful tool for the efficient implementation of a project. The project lacked an operations plan that could have settled many of the ensuing confusion about process flows, procedures, and other technical details in the implementation of the project. An institutional plan could have also clarified the roles, responsibilities and powers of the bodies created to implement the operation. The execution of the project suffered from many implementing entities having ill-defined roles and powers. The Steering Committee turned out to have been prohibited by law from enacting governmental decisions because it included representatives from the academic community, civil society, and other non-governmental bodies. The Advisory Committee could advance recommendations but not make binding decisions. And in this case, the PIU did not have budget executions powers. In view of the variances between Bank procurement standards and guidelines and a borrower’s procurement conventions and practices, it is vital that the Bank and a government thoroughly vet and settle differences in procurement policies and methods before project implementation. It is also vital that the Bank determine that a client government has a full understanding of the project procurement rules. The Loan Agreement (pages 12-14) required adherence to the Bank’s Procurement Guidelines and Consultant Guidelines. Nonetheless, procurement was a huge challenge for this operation because of "policies that conflicted with each other", "confusion regarding the application of which policies", and a "lack of a thorough understanding of the procurement process". Where child labor is a known practice, the Bank and a government need to investigate and discuss the issue early on during project preparation. Children work in post-harvest activities in the export vegetable value chain in rural Guatemala, a practice common among indigenous communities in Latin America, where the family acts as a labor unit. The issue was raised with the Bank during project implementation. In response, the Page 22 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GT Enhancing MSME Productivity Project (P112011) Bank barred disbursements from the second component of the project until mitigation measures were developed, an action that delayed implementation of the component by half a year. Agreement was subsequently reached on a suitable course of action: (a) grant sub-agreements committed the project beneficiaries to desist from using child labor; (b) value chain training activities were to include training on child labor issues; (c) value chain social assessments were to cover child labor issues; and, (d) the Government pledged to protect the rights of children to security, education, and recreation. Restructuring exercises need to focus on actions that are germane to the objectives of the project and not be distracted by unrelated or peripheral proposals, particularly near a project closing date. A Government request for the funding of feasibility studies for irrigation projects lay outside the scope of the project. Similarly, extraneous to the project were attempts to incorporate activities targeted at the communities affected by the construction of the Chixoy hydroelectric dam four decades earlier. While providing individual and collective reparations to displaced communities is a priority for the Guatemalan Government, these proposals would have been better considered as part of a new operation, with adequate time and resources provided for their preparation, rather than being tacked on to an old project nearing its closing date. 14. Assessment Recommended? No 15. Comments on Quality of ICR The strengths of the ICR are: The ICR presents a complete record of the project. It documents the first and the second restructuring of the project comprehensively, detailing which activities remained active and which had been dropped and recording how the costs of the project components changed (ICR, pages 3-4). It tracks changes with the results indicators succinctly, summarizing the revisions in a simple table (ICR, pages 35-36). The document itself is concise, consisting of 23 pages, with the details of the project activities and outputs, including the value chain sub-projects, maintained in an annex (ICR, pages 25-31). The analysis of the performance of the project is evidence-based. Because the scale and scope of the project was significantly revised over two restructurings, the ICR carefully lays out a detailed comparative analysis of the original and revised project, covering --- the PDO (pages 3-4), the components (pages 4-6), the Results Framework (pages 8, 10-11), the M&E plan and implementation (pages 11-12), and the efficacy of the operation (pages 15-21). In all cases, the analysis is candid. The ICR is concise, covering all aspects of the project as required by the OPCS guidelines in a main text of under 30 pages. Annex 2 (Outputs by Component) is especially useful, providing a tally of the key activities and outputs of the project, organized by component and sub-component. Page 23 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GT Enhancing MSME Productivity Project (P112011) a. Quality of ICR Rating Substantial Page 24 of 24