The World Bank CLEAR - Brazil (P150738) REPORT NO.: RES33164 RESTRUCTURING PAPER ON A PROPOSED PROJECT RESTRUCTURING OF P150738 - CLEAR - BRAZIL APPROVED ON OCTOBER 20, 2015 TO Fundação Getulio Vargas São Paulo School of Economics (FGV-EESP) Regional Vice President: Jorge Familiar Calderon Country Director: Martin Raiser Senior Global Practice Director: Sophie Sirtaine Practice Manager/Manager: Sophie Sirtaine Task Team Leader: R Jason Thomas Mayfield The World Bank CLEAR - Brazil (P150738) ABBREVIATIONS AND ACRONYMS CLEAR Centers for Learning on Evaluation and Results ECD Evaluation Capacity Development EESP São Paulo School of Economics FGV Fundação Getulio Vargas LEAP Learning from Performance MDTF Multi-Donor Trust Fund M&E Monitoring & Evaluation PDO Project Development Objective The World Bank CLEAR - Brazil (P150738) BASIC DATA Product Information Project ID Financing Instrument P150738 Investment Project Financing Original EA Category Current EA Category Approval Date Current Closing Date 20-Oct-2015 31-Dec-2018 Organizations Borrower Responsible Agency FGV-EESP FGV-CLEAR Financing (in USD Million) FIN_SUMM_PUB_TBL SUMMARY Total Project Cost 0.20 Total Financing 0.00 Financing Gap 0.00 DETAILS -NewFin3 Project Development Objective (PDO) Original PDO The objective for the project is to contribute to monitoring and evaluation information being increasingly used to make policy and program decision and to strengthen Brazil's monitoring and evaluation systems. OPS_TABLE_PDO_CURRENTPDO The World Bank CLEAR - Brazil (P150738) Summary Status of Financing Net TF Approval Signing Effectiveness Closing Commitment Disbursed Undisbursed TF-A1125 20-Oct-2015 30-Oct-2015 30-Oct-2015 31-Dec-2018 .28 .28 0 Policy Waiver(s) Does this restructuring trigger the need for any policy waiver(s)? No I. PROJECT STATUS AND RATIONALE FOR RESTRUCTURING A. Background The Multi-Donor Trust Fund (MDTF) – Centers for Learning on Evaluation and Results (known as the “CLEAR Initiative”) - is a roughly US$15 million programmatic MDTF established in 2009 to develop global monitoring and evaluation (M&E) capacities, with an additional $2M in contributions expected. The CLEAR Initiative has regional centers located in Mexico, Brazil, South Africa, Senegal, China and India that work to develop local M&E capacities. The program’s website is www.theclearinitiative.org and its most recent annual report is at https://www.theclearinitiative.org/sites/clearinitiative/files/2018-01/CLEAR_Annual_Report_2016-2017.pdf. CLEAR began giving grants in 2011 to its regional centers, selected competitively and housed in academic institutions. They have the mandate to deliver and to help replicate effective M&E capacity development services. The centers offer a menu of products and services to strategic partners and clients in their regions, including government, civil society, the private sector, and other M&E capacity building providers. Among the range of capacity development products and services provided are technical assistance to build client M&E systems, knowledge and research services, and training. The São Paulo School of Economics (EESP) at the Fundação Getúlio Vargas (FGV) in Brazil was competitively selected in 2015 to implement the CLEAR program and to establish the CLEAR Brazil and Lusophone Africa Center (FGV/EESP CLEAR). FGV, based in São Paulo, Brazil, is a well-renowned think-tank and higher education institute dedicated to promoting Brazil’s economic and social development. FGV/EESP CLEAR started operations in September 2015. Their current grant (P150738 of $277,000), which was approved in October 2015 through June 2018 (with an extension granted until December 2018), allows FGV/EESP CLEAR to establish strong relationships with public sector and corporate clients and build the foundations of its future work at all levels of evaluation capacity development. The World Bank CLEAR - Brazil (P150738) B. Project Status Since the beginning of the current grant period, FGV/EESP CLEAR has been on a solid track towards achieving its project development objectives. The Center has been able to leverage additional funds for the project; both from client fees and fundraising from donors. Through its portfolio of trainings, workshops, technical advisory and knowledge sharing services, FGV/EESP CLEAR has supported the M&E capacity development for a diverse set of strategically selected clients from government agencies and corporate social branches of private sector organizations. The key activities the Center is pursuing under the current grant (TF0A1125/ P150738) are: (i) improving M&E practices and increasing the uptake of evidence in decision making by providing technical assistance to public sector clients (national and subnational level) aware of the importance of M&E; (ii) improving M&E practices and knowledge in corporate social branches by providing technical assistance on M&E systems and generating evidence through evaluations; (iii) training courses to strengthen M&E skills and knowledge of individuals and organizations in its region; and (iv) outreach activities to raise awareness to the importance of evidence-based policy making. The total number of client capacity building activities completed in FY16 and FY17 is 42, including 9 trainings or workshops reaching over 790 participants and 19 advisory services. The participants from FGV/EESP CLEAR trainings and workshops have consistently rated the quality and usefulness of its services as “high” (that is, “4 or 5” on a 5-point scale). C. Progress on PDOs Indicator Name FY16 FY17 Target Net revenue is positive for at least 50% of center activities by year 2018 for advisory 60% 50% 50% services (fee-charging) Comment: For the provision of advisory services, the center either charges fees from clients or receives grants from third party donors - that is, who are funding the provision of services to a (usually public) organization. Since these services are the Center’s main source of net income, the 50% positive revenue target is integral for its long term financial sustainability. 85% of the Centers’ client rate the quality of their services highly, as measured 85% 85% 85% through feedback forms after the delivery of service Comment: The Center started collecting information on this indicator in FY2017, once it had been able to establish itself and set up internal monitoring procedures. Rates refer to feedback forms received upon the completion of seminars, workshops and courses, from which the Center has gained important knowledge on targeting basic, The World Bank CLEAR - Brazil (P150738) intermediate and advanced content to various types of audiences and as improving its general communication strategy. In FY2017, 85% of participants rated the general quality of the Center’s activities as 4 or 5 on a five-point scale. These activities have addressed a wide variety of audiences. Seminars covered specific topics - such as the establishment of M&E systems and the evaluation of technical education in Brazilian states - and were open to guests and public alike. Workshops introduced participants from governments, public agencies, nonprofit organizations and universities to M&E tools. Courses, in turn, have so far been customized and delivered to client organizations. 75% of the Centers clients rate the usefulness of their services highly, as measured N/A N/A 75% through tracer surveys after the delivery of the service Comment: Tracer surveys are collected for courses and technical assistance services at least 6 months after their completion. The Center took on a few long technical assistance projects which were only concluded in FY2017, and their respective tracer studies are currently in process. The Center also offered its first courses in FY2017, and tracer surveys have been sent out with initial results being available in FY2019. D. Rationale for Restructuring The Administration Agreements for the program and ‘parent’ trust fund were recently extended from 2018 to June 30th, 2021. The Funding Committee of donors (the Board) of the CLEAR multi-donor trust fund (MDTF) program, of which the Secretariat (or Global Hub) sits in the World Bank Group’s Independent Evaluation Group, concluded in a meeting (in October 2017) that the grant given to the FGV/EESP CLEAR ending in December 2018 should be extended with additional financing, until the new end date of the CLEAR global program. Given the strong performance of the Center to date, a recent mission to FGV/EESP CLEAR (February 2018) also concluded that the grant date should be extended with additional financing to allow the Center to continue delivering evaluation capacity development services in Brazil until the end of the fund in 2021, and to keep momentum on the progress made towards financial sustainability. There is still significant unmet demand in the region that warrants the supply of CLEAR’s services. Taking into consideration the performance of the Center verified by the mission and the recommendation by the Funding Committee, there is strong rationale to restructure the current grant. An approval is thus sought for a two-year extension of the closing date to December 30, 2020 for Grant TF0A1125 (P150738), including additional financing to the Recipient amounting to US$500,000 via Grant TF0A8518, as recommended by the donors to CLEAR. The proposed extension and additional financing will allow FGV/EESP CLEAR to continue and intensify its work on supporting the M&E capacity development for a diverse set of strategically selected clients. The World Bank CLEAR - Brazil (P150738) II. DESCRIPTION OF PROPOSED CHANGES A. Project Development Objective The Project Development Objective is being revised to: “The objective of the project is three-fold: (i) strategic clients gain knowledge in, and the motivation to use M&E methods, approaches, tools, and findings; (ii) other M&E capacity providers gain knowledge on evaluation capacity development (ECD) from CLEAR’s experience on what works and what doesn’t in ECD; and (iii) the CLEAR Center works towards achieving institutional and financial viability.” This brings the PDO agreement with the other five center grants in the CLEAR network and will facilitate measurement of results as a network. B. Closing Date The revised grant closing date of December 31st, 2018, is being extended to December 30, 2020. The multi-donor trust fund closing date is June 30th, 2021. The closing date of December 31, 2020 for the FGV/EESP CLEAR grant will provide ample time to finish all necessary processing in time for the closing of the fund. C. Additional Financing Additional funding amounting to US$500,000. Thus far, on average, the Center has received roughly US$100,000 per year through the current grant period, from late 2015 through 2018. The additional amount of US$ 500,000 will cover the period from approval of additional financing through December 2020. The components listed below mirror the components from the original grant paper, as do the activities. The additional financing will allow the recipient to scale-up and do an expanded number of the same types of activities for anticipated broader reach and effect. D. Components and Cost There are no changes to the description of the components in the Grant Agreement. While the broad components remain the same, activities within them have been update to: (i) scale-up the ongoing activities proposed to be implemented during the extended period of time (closing date December 2020); and (ii) address regional changes in demand and supply from when the activities were last updated in 2015. 1. COMPONENT 1: Center's Capacity Building Activities The Center’s main objective in component one is to ensure that internal capacity and visibility of the Center is strengthened. This includes ensuring that the Center staff is adequately positioned within Brazil, integrated into the CLEAR Global community and other M&E networks. To meet the objectives of Component 1, the Center will engage in a variety of internal capacity building activities, which will help develop its management, administrative and professional capacities, including the continuation and/or scaling up of the following activities: The World Bank CLEAR - Brazil (P150738) (a) Strengthening internal capacities by hiring consultants to perform financial, legal and strategic consulting services, as well as providing the Center’s staff with internal and occasionally external training to fill in existing gaps – i.e. hiring a consultant to develop a fundraising strategy for the Center and train the staff; (b) Engaging with the CLEAR Initiative’s global network through participation in activities such as the annual CLEAR Global Forum, CLEAR panels in M&E network conferences and regional centers’ knowledge-sharing webinars, as well as joint development of knowledge products and delivery of CLEAR services; (c) Carrying out periodic Learning from Performance Reviews (LEAP), along with IEG staff, to ensure that there is continuous learning from the center’s activities and that the program’s strategic focus remains on track to achieve the overall CLEAR goals; (d) Maintaining the Center’s structure and investing in the Center’s communication activities and materials, including institutional stationery, printed publications and digital marketing. 2. COMPONENT 2: Capacity Development Services for Clients The objective of Component 2 remains the same, providing capacity development services for clients, and, at the same time, developing and sharing knowledge/learning on the “how to” of building M&E capacity, thus contributing to CLEAR’s global knowledge and learning generation. One of the main lessons learned with the work performed so far has been that prioritizing existing demand for ECD services, which comes mainly from organizations and networks with at least some level of awareness of the importance of M&E, generates the greatest impact vis-à-vis the Center’s small staff and limited time. Therefore, activities under this component will continue to contemplate raising awareness, but greater focus will be placed on training courses, research, advisory and evaluation services to strategically selected clients, with a multiplying effect in Brazil. (a) Raising individuals’ level of M&E knowledge and skills: (i) Raising M&E Awareness: The Center will continue to offer seminars, participate in external events and generate knowledge products with the purpose of raising awareness of the importance of M&E in Brazil. The partnership with the LAPP evidence repository for the production of thematic policy briefs in Portuguese will be maintained, as well as the development of the Center’s own publications on M&E tools, methods and lessons learned on ECD – all of which will be available to the general public. Considerable resources and efforts will be put towards these knowledge products, aiming to consolidate the center’s role as a Brazilian reference in the field for governments, the private sector and academia; (ii) Open-Enrollment Courses: The Center has so far developed materials for two courses - Qualification on M&E (ForMA) and Advanced Qualification on Quantitative Methods for the Evaluation of Public Policies and Social Programs - to be delivered to general audiences on an open-enrollment basis. Despite having encountered some difficulty with this audience so far, the center intends to continue to work towards building capacities on the individual level - mainly through the offering of short courses, ranging between 20 and 35 hours - for which adequate marketing strategies will be developed. Additional material may also be generated as need arises. Offerings will be completely financed by participants’ fees. (b) Strengthening public and private organizational capacities to produce and/or use evaluative evidence, thus contributing to more widespread M&E practices in Brazil: The World Bank CLEAR - Brazil (P150738) (i) Technical Assistance: The Center will continue to provide technical assistance to client organizations in carrying out diagnoses, evaluations and cost benefit analyses for their programs, as well as in designing and implementing M&E strategies. This type of service contributes to increasing evidence-based decision making in Brazil and enables organizations to learn in practice by working with CLEAR’s evaluators. Activities under this subcomponent are completely financed by client fees or other grants received for the performance of specific services, and often generate net income for the Center; (ii) Training Client Organization’s Staff: In 2017 the Center customized its 138-hour long introductory course, Qualification on M&E (ForMA), for delivery to public officials from two state governments, São Paulo and Espírito Santo. This has proved to be a very fruitful line of work, securing longer relationships with the clients and enabling the Center to deepen its subnational agenda. The Center will, therefore, look to expand it to other subnational governments, including states and municipalities, which may require subsidizing. (c) Engaging institutions and sectors around the M&E agenda, promoting an enabling environment for the production and use of evidence in policy making (i) Introductory M&E Trainings: The Center will continue to provide workshops to introduce representatives from specific groups or sectors to M&E. These short trainings aim to raise participants’ awareness of the importance and usefulness of basic M&E tools as a way of engaging more than the individuals attending, but rather the governments, associations and networks they represent or are connected to. The Center will look to establish more partnerships to expand the delivery of such trainings, especially in Brazil’s least developed regions; (ii) Assistance for M&E Institutionalization: The Center will continue advancing its relationship with the Brazilian Federal Government, from which a technical assistance for the design of a new M&E system has already stemmed. Similar advisory services have been provided to the State of Espírito Santo, which has become the first in the country to pass legislation enforcing M&E practices. Both experiences have not only strengthened organizational capacities, but also brought a range of departments and agencies together around the M&E agenda. This background enables the Center to increase the effectiveness of this sort of service and provides successful precedents for other governments, thus favoring the Center’s pursuit of the subnational M&E agenda. Fees for service allow full cost recovery and occasional positive income for these types of service; (iii) Targeted Knowledge Sharing: The Center will continue to share knowledge on general M&E tools and lessons and thematic policy evidence by organizing and participating in seminars with targeted audiences, thus raising M&E awareness on a broader environment. It also currently produces a large amount of applied research and general knowledge products that are released on an individual-level basis, that is to say, not targeting specific audiences that could benefit from them. Going forward, it intends to use existing and new products to further engage institutions and networks in the debate about M&E practices and thematic evidence. This effort will entail substantial resources for dissemination and communication activities; (iv) Interorganizational Policy Evaluations: The Center will continue to perform evaluations of policies that involve or affect a range of institutions. These services are completely financed by other grants and/or as academic research, and the results found contribute to strengthening the use of evidence for policy, supporting the generation of an environment for M&E. 3. COMPONENT 3: Monitoring and Evaluation of the CLEAR activities The World Bank CLEAR - Brazil (P150738) This component will focus on developing a monitoring and evaluation report system for the CLEAR work program. The Center will keep regular monitoring information and quality assessment data for each training and advisory service provided. These data, together with a progress report comprising program accomplishments, financial statements, and issues to be addressed will be provided to the CLEAR Global Team at IEG. Specific activities related to this include: (a) Collection of feedback forms from the Center’s seminars and workshops - whenever possible, as some activities are organized within larger conferences or in partnership with other organizations, and form distribution may be unfeasible; (b) Collection of feedback forms for each module and for the Center’s courses as wholes by their end, as well as tracer surveys within six months of their completion; (c) Collection of tracer surveys for the Center’s technical assistance services within at least six months of their completion; (d) Submission of a Monitoring Tool according the CLEAR Global Team’s standards - which includes service quality assessment and financial data - every 6 months. With the additional financing, the cost per components has been revised: Disbursed Amount: October Original Grant (P150738) Additional Financing of Combined Original Grant + 2015- (Until December 2018) $500,000) Additional Financing) February 2018 Disbursed Original Original % of Total Amount % of +Additional % of Total Grant Grant Additional (USD) Additional Grant Grant Project Components Planned Planned Financing Financing Financing Financing Financing Financing Amount Amount Amount (C) Amount Amount Amount + (E) (A) (C) (D) (E) (F) (G) (H) Grant Auditing $0 $0 $0 $ 20,000 4% $ 20,000 2.57% FGV’s Overhead $ 21,794 $ 36,010 13% $ 65,000 13% $101,010 13% Component 1: Capacity Building Activities for CLEAR SA 1. Management and internal capacity $ 7,757 $0 0% $ 100,000 20% $ 100,000 12.87% building 2. Engagement with the CLEAR global $ 39,081 $ 20,893 7.5% $ 47,000 9% $ 67,893 8.74% network 3. Strategy planning $0 $0 0% $0 0% $0 0% through LEAP reviews 4. Structure and $ 9,805 $ 5,142 16.2% $ 10,000 2% $ 15,142 1.95% communication The World Bank CLEAR - Brazil (P150738) Component 2: Capacity Development Services for Clients A. Raising the level of knowledge and skills in M&E capacity of ‘individuals’ 1. Raising M&E $ 39,767 $ 58,243 21% $ 45,000 9% $ 103,243 $ 13.29% Awareness 2. Open Enrollment $ 32,278 $ 66,286 23.9% $ 10,000 2% $ 76,286 $ 9.82% Courses B. Strengthening public and private organizational capacities to produce and/or use evaluative evidence, thus contributing to more widespread M&E practices in Brazil 1. Technical Assistance $0 $0 0% $0 0% $0 0% 2. Training Client $ 4,968 $0 0% $ 38,000 8% $ 38,000 4.89% Organization’s Staff C. Engaging institutions and sectors around the M&E agenda, promoting an enabling environment for the production and use of evidence in policy making 1. Introductory M&E $ 9,126 $0 0% $ 10,000 2% $ 10,000 1.29% Trainings 2. Assistance for M&E $ 12,468 $0 0% $ 20,000 4% $ 20,000 2.57% Institutionalization 3. Targeted Knowledge $ 2,892 $0 0% $ 75,000 15% $ 75,000 9.65% Sharing 4. Interorganizational $ 9,156 $ 52,710 19% $0 0% $ 52,710 6.78% Policy Evaluations Component 3: Monitoring and Evaluation of CLEAR Activities 1. Feedback for seminars and $ 12,576 $ 9,429 3.3% $ 15,000 3% $ 24,429 3.14% workshops 2. Feedback and tracer $ 12,576 $ 9,429 3.3% $ 15,000 3% $ 24,429 3.14% studies for courses 3. Tracer studies for technical assistance $ 12,576 $ 9,429 3.3% $ 15,000 3% $ 24,429 3.14% services The World Bank CLEAR - Brazil (P150738) 4. Monitoring Tool $ 12,576 $ 9,429 3.3% $ 15,000 3% $ 24,429 3.14% TOTAL COST $ 239,396 $ 277,000 100% $ 500,000 100% $ 777,000 100% E. Sustainability of the Center Taking calendar years 2016 and 2017 into account, CLEAR grant funds correspond to about 23% of the Center’s revenue. Some of the challenges around sustainability include a low willingness to pay from some government partners and from individual professionals, as well as limited staff capacity. From the outset, the Center has not been overly dependent on the CLEAR grant, and most of its sustainability strategy relies on maintaining income-generating lines of work while seeking some complementary sources of funding: - Generating net income from technical assistance and advisory services to clients with high willingness to pay, even if such services and clients are not highly strategic, in order to cross-subsidize important projects in the absence of CLEAR grant funds; - Fundraise with national and international donors to fund evaluation research in thematic areas, with a focus on large grants obtained in partnership with other academic institutions and research centers. The different aspects of this strategy are summarized below.: 1. Keeping financially rewarding technical assistance and advisory services: The Center has been working with private sector clients (foundations and corporate social responsibility branches), whose willingness to pay is high, by providing technical assistance and advisory services. Some of these clients and/or projects are not the most strategic vis-à-vis the impact the Center aims to generate on evidence-based policy making in Brazil, but remain the Center’s main source of net income, which is important not only for its overall sustainability, but also in subsidizing some of its most strategic work with the public sector. In this regard, partnerships with FGV PROJETOS, the Center’s host institution’s consulting branch, should also be intensified; 2. Fundraising with donors to fund evaluation research in thematic areas: To be able to fund thematic research in the Center’s areas of expertise, thus generating important evidence to fill in gaps in the Brazilian context and ultimately influence policy making, more national and international donors should be sought, including government agencies, private foundations and international organizations. Funds will be sought in partnership with other research centers in FGV and other academic institutions from Brazil and abroad – enhancing the Center’s eligibility for different grants. A fundraising specialist consultant should also be hired to assist the Center in setting up a specific strategy and training its staff; 3. Consolidating introductory and advanced trainings: Trainings, be they open-enrollment or customized for a client organization’s staff, are not a major source of income for the Center. However, they are important way to retain a portion of the staff, who receive low fixed salaries and complement their income by teaching classes in CLEAR trainings. Additionally, trainings are also an important entry point for establishing longer relationships with strategic clients, which may later lead to paying technical assistances; 4. Expanding partnerships for awareness raising events: The Center has been partnering with different organizations and networks to be able to throw events aiming to raise awareness of the uses and importance of M&E by only providing in-kind resources – staff time and, occasionally, space at FGV. This is a sustainable way to contribute to The World Bank CLEAR - Brazil (P150738) the Center’s mandate to propagate evidence-based policy making in Brazil and achieve more visibility for its brand, and partnerships should be expanded. The World Bank CLEAR - Brazil (P150738) A. Results Framework Baseline Target Values Responsi Data Source/ bility for Description Indicator Core Unit Frequency Methodology Data (indicator definition, etc) 2017 Yr 11 Yr 22 Yr 33 Yr 44 Collection PDO-Level Result Indicators Indicator One: % of Center’s strategic This information is clients who rate “increase in Measured by collected after training knowledge and/or skills” as end of training courses and workshops. a result of participating in its courses or Feedback 83% 85 90 90 90 Center Standard CLEAR courses and workshops Yes % workshops and Forms feedback surveys highly (4 or 5 on five-point reported every available, but can be scale), as measured through 6 months modified by the Center feedback forms after delivery of service Indicator Two: % of Center’s strategic clients who rate the extent This information is to which they applied collected through tracer Measured concepts learned (for surveys 6 months after 6 months after training services) or adopted Tracer service completion. Yes N/A 70 75 80 85 service and Center recommendations (for % Surveys Standard CLEAR tracer reported advisory services) to make surveys available, but annually changes in their current can be modified by the work as high or significant, Center as measured through tracer surveys Indicator Three: Number of knowledge products Targeted at providers of generated, including Monitoring M&E services, Yes Number 6 7 4 6 4 Annually Center publications, policy briefs Tool practitioners, scholars and technical assistance and students. reports Indicator Four: Proportion Yes % 2.93% 21% 25% 20% 10% Annually Monitoring Center of CLEAR MDTF revenue to Tool total revenue (including 1 July 2017 - June 2018 CLEAR MDTF and all other 2 July 2018 sources, such as June - fees for2019 3 July contributions service, 2019 - June from 2020 host institutions 4 June 2020 - and Decother 2020 The World Bank CLEAR - Brazil (P150738) grants) Intermediate Outcome and Output Indicators % of Center’s advisory This information is clients who rate the extent collected through tracer to which the overall Measured surveys 6 months after performance of their 6 months after Tracer service completion. organization/unit/program No % N/A 50 60 70 80 service and Center Surveys Standard CLEAR tracer has improved as a result of reported surveys available, but CLEAR services as high or annually can be modified by the significant, as measured Center through tracer studies Measured This information is % of Center clients who rate by end of collected after training the “quality” of training and training courses and workshops. workshop services highly (4 Feedback No % 85% 85 90 90 90 courses or Center Standard CLEAR or 5 on five-point scale), as Forms workshops and feedback surveys measured through feedback reported every available, but can be forms 6 months modified by the Center This information is % of Center clients who rate Measured collected after training the "usefulness" of training by end of courses and workshops. and workshop services Feedback No % 84% 85 90 90 90 training Center Standard CLEAR highly (4 or 5 on five-point Forms courses or feedback surveys scale), as measured through workshops available, but can be feedback forms modified the by Center Fees collected from open- Targeted at individual enrollment and Reported every Monitoring No USD $0 $38,010 50000 50000 50000 Center professionals and client organizational/customized 6 months Tool organizations courses Targeted at client Fees collected from Reported every Monitoring No USD $762,661 $47,266 200000 450000 600000 Center organizations, both technical assistance services 6 months Tool public and private Targeted at individual Reported every Monitoring Number of training courses. No Number 0 2 3 4 2 Center professionals and client 6 months Tool organizations Targeted at individual Number of training Reported every Monitoring No Number 4 5 4 8 6 Center professionals and client workshops 6 months Tool organizations Number of technical No Number 7 2 4 6 4 Reported every Monitoring Center Targeted at assistance services 6 months Tool organizational clients such as government, line The World Bank CLEAR - Brazil (P150738) ministries & subnational governments Targeted at providers of M&E services, Number of knowledge Reported every Monitoring practitioners, academia, No Number 6 6 4 6 4 Center sharing activities 6 months Tool governments, public agencies, nonprofits and private foundations Measured through Total number of participants Monitoring No Number 0 80 105 140 70 registration in Center in training courses Tool training courses Measured Total number of female by end of Monitoring participants in training No Number 0 39 50 70 35 training Center Tool courses courses or workshops Monitoring Tools delivered Reported every Monitoring No Number 2 2 2 2 2 Center on time to the CLEAR Hub six months Tool Measured Tracer surveys administered six months Monitoring for training courses and No Number 0 8 11 18 12 Center after service Tool advisory services completion The World Bank CLEAR - Brazil (P150738) B. Procurement There are no changes in the procurement conditions or arrangements. As part of the annual procurement review and for the additional financing preparations, the procurement plan and procurement schedule were updated and approved on May 23rd, 2018 for a period of 24 months. The primary updates included: (a) procurement of goods, consulting services, and trainings and workshops - the procurement methods and thresholds, including prior review thresholds; and (b) selection of consultants - the selection methods and thresholds, including the prior review threshold. C. Financial Management Financial Management and Disbursement Arrangements currently in place will remain the same, with one exception related to the external auditing arrangements. FGV has sufficient assurance mechanisms to guarantee the proper use of funds, and given the low-risk profile of the Grant, a single audit report to cover the entire life of the extension, rather than annual audited financial statements, has been approved. The report (and any accompanying management letter) should be submitted to the Bank no more than six months after the Closing Date. D. Safeguards Policies There are no changes in the safeguard policies that may apply. No safeguard policies were triggered in the original grant agreement. No appraisal stage ISDS was required. III. SUMMARY OF CHANGES Changed Not Changed Project's Development Objectives ✔ Results Framework ✔ Loan Closing Date(s) ✔ Additional Financing Proposed ✔ Implementing Agency ✔ Components and Cost ✔ Cancellations Proposed ✔ Reallocation between Disbursement Categories ✔ Disbursements Arrangements ✔ Disbursement Estimates ✔ Change in Overall Risk Rating ✔ The World Bank CLEAR - Brazil (P150738) Safeguard Policies Triggered ✔ EA category ✔ Legal Covenants ✔ Institutional Arrangements ✔ Financial Management ✔ Procurement ✔ Implementation Schedule ✔ Other Change(s) ✔ IV. DETAILED CHANGE(S) OPS_DETAILEDCHANGES_PDO_TABLE PROJECT DEVELOPMENT OBJECTIVE Current PDO The objective for the project is to contribute to monitoring and evaluation information being increasingly used to make policy and program decision and to strengthen Brazil's monitoring and evaluation systems. Proposed New PDO The objective of the project is three-fold: (i) strategic clients gain knowledge in, and the motivation to use M&E methods, approaches, tools, and findings; (ii) other M&E capacity providers gain knowledge on evaluation capacity development (ECD) from CLEAR’s experience on what works and what doesn’t in ECD; and (iii) the CLEAR Center works towards achieving institutional and financial viability. OPS_DETAILEDCHANGES_RESULTS_TABLE RESULTS FRAMEWORK Project Development Objective Indicators PDO_IND_TABLE Proportion of CLEAR MDTF revenue to total revenue (including CLEAR MDTF and all other sources, such as fees for service, contributions from host institutions and other grants) Unit of Measure: Percentage Indicator Type: Custom Baseline Actual (Current) End Target Action Value 2.93 5.00 20.00 Revised Date 30-Oct-2015 15-Dec-2016 31-Dec-2020 % of Center's strategic clients who rate "increase in knowledge and/or skills" as a result of participating in its courses and workshops highly (4 or 5 on five-point scale), as measures through feedba Unit of Measure: Percentage The World Bank CLEAR - Brazil (P150738) Indicator Type: Custom Baseline Actual (Current) End Target Action Value 83.00 85.00 90.00 Revised Date 30-Oct-2015 15-Dec-2016 31-Dec-2020 % of Center's strategic clients who rate the extent to which they applied concepts learned (for training services) or adopted recommendations (for advisory services) to make changes in their current w Unit of Measure: Percentage Indicator Type: Custom Baseline Actual (Current) End Target Action Value 0.00 85.00 Revised Date 30-Oct-2015 31-Dec-2020 Cumulative number of knowledge products generated, including publications, policy briefs and technical assistance reports Unit of Measure: Number Indicator Type: Custom Baseline Actual (Current) End Target Action Value 6.00 7.00 21.00 New Date 01-Jul-2017 31-Dec-2020 Intermediate Indicators IO_IND_TABLE Annual ‘Learning from Performance Review' of Center activities with clear decisions Unit of Measure: Number Indicator Type: Custom Baseline Actual (Current) End Target Action Value 0.00 1.00 2.00 No Change Date 30-Oct-2015 30-Jun-2016 01-Jun-2018 OPS_DETAILEDCHANGES_LOANCLOSING_TABLE The World Bank CLEAR - Brazil (P150738) LOAN CLOSING DATE(S) Original Revised Proposed Proposed Deadline TF Status Closing Closing(s) Closing for Withdrawal Applications TF-A1125 Effective 01-Jun-2018 31-Dec-2018 31-Dec-2020 30-Apr-2021 OPS_DETAILEDCHANGES_ADDITIONAL_FINANCING_TABLE ADDITIONAL FINANCING Source Currency Amount USD Equivalent ECD FOR REGIONAL CENTERS(ECD) USD 500,000.00 500,000.00 Existing Net Commitment USD Amount 280,000.00 Total 780,000.00