Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Bolsa Familia 2nd APL (P101504) Report Number: ICRR0021905 1. Project Data Project ID Project Name P101504 BR Bolsa Familia 2nd APL Country Practice Area(Lead) Brazil Social Protection & Jobs L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-78410 30-Dec-2015 196,160,814.42 Bank Approval Date Closing Date (Actual) 16-Sep-2010 31-Dec-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 200,000,000.00 0.00 Revised Commitment 196,160,814.42 0.00 Actual 196,160,814.42 0.00 Prepared by Reviewed by ICR Review Coordinator Group Salim J. Habayeb Judyth L. Twigg Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives According to the Loan Agreement of 9/21/11, the objective of the project was to strengthen the Bolsa Familia Program’s (BFP) ability to achieve its objective of reducing poverty and inequality and promoting human capital development by improving schooling and health status of children and reducing incidence of malnutrition among the poor population. (Note: Bolsa Familia Program means Family Grant Program.) Page 1 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Bolsa Familia 2nd APL (P101504) The project constituted a second phase of an Adaptable Program Loan (APL2) which followed a first phase (APL1) that was undertaken during 2004–2009. APL2 development objectives conformed to the objective originally described for the APL, i.e., strengthening the program’s ability to reduce poverty and inequality by providing poor families with income transfers and incentives to invest in the human capital of their children (Project Appraisal Document, PAD, p. 4). In terms of APL phasing, the Bank supported the Government of Brazil in establishing the conceptual, legal, and institutional foundations of BFP under APL1 (PAD, pp. 4-5), while APL2 sought to further strengthen and consolidate the safety net and to deepen BFP’s technical improvements and promote the links of BFP beneficiaries to other services. During implementation, the project expanded the scope of its activities, but one associated outcome target was revised downwards in 2017 because of methodological changes in data collection and issues of comparability among related surveys (ICR, pp. 12-13). At the time of the target revision, the disbursement rate was 98%. A split evaluation would result in a negligible weight value after the revision and would not impact the overall outcome assessment. Therefore, this ICR Review did not apply a split evaluation. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? No c. Will a split evaluation be undertaken? No d. Components 1. Conditional cash transfers (Appraisal US$10 billion of which IBRD was US$185 million; Actual IBRD US$185 million). Provision of grants to mothers or other designated family members to enable their families to access health, education, and other services as provided in the BFP. 2. Strengthening the Cadastro Único (single registry) as the main targeting instrument for social programs (Appraisal US$6 million of which IBRD was US$2.8 million; Actual IBRD US$2.6 million). Page 2 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Bolsa Familia 2nd APL (P101504) Provision of support, training, incremental operating costs, and technical assistance to improve and consolidate the Cadastro Único as the main mechanism to select the target population and to integrate social programs at various levels. This would include improving information quality, strengthening the identification of Indigenous Peoples, strengthening information technology and systems, promoting access and use of the single registry for decision-making by other programs and government agencies, and supporting surveillance on living conditions of low-income families. 3. Institutional strengthening for the consolidation of BFP (Appraisal US$9.7 million of which IBRD was US$4.6 million; Actual IBRD US$4.30 million). Strengthening management, governance, and accountability capacity of the Ministry of Social Development (MDS) to implement BFP, including through: (a) Strengthening BFP overall management and operation, including strengthening decentralized management, consolidating control and accountability mechanisms, implementing a capacity building strategy to enhance the capacities of program staff and control agencies at federal and sub-national levels, and provision of information technology equipment. (b) Improving management of BFP co-responsibilities, including improving compliance verification, strengthening social control mechanisms, and consolidating the Good Practices Observatory and Contest. (c) Improving management of benefits, including updating socio-economic data on beneficiaries, improving targeting, strengthening the strategy of opening accounts for beneficiaries and payments in remote areas, developing instruments of financial education, and improving distribution of electronic payment cards. 4. Consolidation of the monitoring and evaluation system of the MDS (Appraisal US$9.3 million of which IBRD was US$4.5 million; Actual IBRD US$2.6 million). Provision of support, training, technical assistance, equipment, and selected incremental operating costs to consolidate MDS capacity to monitor and evaluate social programs at the federal and sub-national levels. 5. Supporting the National Commitment for Social Development (CNDS) (Appraisal US$2.9 million of which IBRD was US$1.3 million; Actual IBRD US$1.2 million). Page 3 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Bolsa Familia 2nd APL (P101504) Provision of support, training, selected incremental operating costs, and technical assistance to MDS to make CNDS operational and to facilitate social program integration. 6. Operational support to the Project Implementation Unit at MDS (Appraisal US$1.7 million of which IBRD was US$1.2 million; Actual IBRD US$0.3 million). Component revisions The first project restructuring in 2014 reallocated activities within components to reflect the new structure of MDS that took effect with the launch of a new program: Brazil without Poverty Plan (PBSM), and whose objective was to increase capacities and opportunities for the extreme poor registered in Cadastro Único to increase their income and improve basic social well-being. The second restructuring in 2017 added new technical assistance activities, including the design of an early childhood development program (Criança Feliz) for BFP beneficiaries, evaluation of key MDS programs that did not have a recent evaluation, upgrading of the single registry to further improve targeting capabilities and coordination with other social programs, and adjustment of the institutional framework for implementation to reflect another MDS restructuring following a presidential administration change (ICR, pp. 9-10). e. Comments on Project Cost, Financing, Borrower Contribution, and Dates APL2 was designed as a Sector-Wide Approach or SWAp whereby the Bank contributed to the larger government program, estimated at US$10.2 billion under the project (ICR, p. 6), with a focus on supporting technical assistance. The original amount of IBRD financing was estimated at US$200 million, and the actual amount was US$196.2 million. The project was designed for a 5-year implementation period. It was approved on 9/16/10 and became effective more than one year later on 11/24/11 largely due to a change in presidential administration. A Mid- term Review was undertaken on 3/24/14. A first level-2 restructuring in 2014 extended the closing date by 18 months, and a second level-2 restructuring on 5/31/17 further extended the closing date by 18 months to allow sufficient time for the full implementation of the technical assistance components, and to advance the new priority social agendas of the Government, as described above under Component Revisions in Section d. The project closed on 12/31/18, three years after the original closing date of 12/30/15. Page 4 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Bolsa Familia 2nd APL (P101504) 3. Relevance of Objectives Rationale At entry, the operation was responsive to country needs and priorities, and was consistent with the Brazilian government strategies to reduce poverty and inequality while fostering social and economic development. Bolsa Família was at the core of Brazil’s social safety net (PAD, p. 32). As stated in Section 2a, APL1 established BFP, while APL2 aimed at strengthening the program and deepening its technical improvements (PAD, p. 4). At the end of APL1, remaining challenges aggregated in two broad areas: (a) further strengthening of the program’s basic architecture, including streamlining processes and operations; and (b) developing stronger links with other social interventions, employment, and skills development opportunities in order to expand the gateway for beneficiary families to escape poverty (PAD, p. 3). At appraisal, the operation was fully consistent with the Bank’s Country Partnership Strategy for 2008-11, namely with its pillar to achieve a more equitable Brazil and the goals of reducing extreme poverty, vulnerability, and social exclusion. At project closing, APL2 was fully consistent with the Bank’s Country Partnership Framework (CPF) for the Federative Republic of Brazil for FY18– FY23. In particular, the project was in line with the CPF’s Objective 1.2 to “increase fiscal sustainability and fairness of pension system and effectiveness of social protection system” under Focus Area 1: “Fiscal consolidation and government effectiveness for sustainable, inclusive and efficient service delivery." The CPF highlighted the need to rationalize and strengthen social assistance programs. It also noted that improved efficiency of social benefit programs required integration of information systems and development of the single registry, as a tool to monitor eligibility and rationalize overlaps in social programs (ICR, p. 11). Under this Objective 1.2, the CPF stated that the Bank would support the government in strengthening its social safety net programs, making them more agile in protecting the most vulnerable, especially by ensuring that the Bolsa Família program benefited the new poor hit by the recent recession. The overall development approach was pertinent, as the objectives sought to contribute to reducing short- term poverty through conditional cash transfers, while concurrently contributing to long-term poverty alleviation by improving human capital among the poor, with a focus on children. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) Page 5 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Bolsa Familia 2nd APL (P101504) EFFICACY_TBL OBJECTIVE 1 Objective Strengthen the ability of Bolsa Familia Program to reduce poverty and inequality. Rationale Although the project development objective was stated in terms of a change in institutional capacity (to “strengthen BFP’s ability to achieve the objective of reducing poverty and inequality and promoting human capital development…”), the ICR’s discussion of the theory of change seemed to assume the objective was actually “BFP’s impact on poverty and inequality reduction, and human capital development.” Despite this shortcoming in the conceptualization of the theory of change specific to the PDO as stated, it was reasonable to expect that improving program functions, management, and oversight; enhancing targeting and registration, improving data quality; promoting linkages with other social programs, facilitating transfer mechanisms to the poor; and improving coverage would lead to increased ability of BFP to contribute to reducing poverty and inequality. Outputs and intermediate results  The project provided conditional cash transfers and technical assistance, including consultancies, studies, evaluations, training, and development of manuals on program operations and payment mechanisms.  The project trained 5,000 persons in Cadastro Unico and BFP management, and 226,000 persons in subnational governments on various operational aspects of the program.  The project supported the launch of Version 7 of the Cadastro Único online system in all municipalities. This updated system was meant to improve information quality, reduce duplication, monitor conditionalities, and enhance governance (ICR, p. 16).  MDS introduced the Decentralized Management Index to transfer resources under performance- based contracts between MDS and municipalities. The synthetic index measured municipal performance comprising variables of coverage, quality of information, and compliance with education and health conditionalities. The average Decentralized Management Index increased from a baseline of 0.80 in 2011 to 0.84 in December 2017, marginally short of the target of 0.85.  The number of beneficiary families increased from a baseline of 12 million in 2009 to 14 million in January 2018, exceeding the target of 12.9 million. This number of beneficiary families consisted of 46.9 million persons, almost a quarter of the total country population.  The percentage of updated records in Cadastro Único version 7.0 increased from a baseline of 40% in 2011 to an almost universal level of 99.7% in December 2017, exceeding the original target of 90% and the revised target of 85%.  The rate of updated registries in the last two years in Cadastro Único increased from a baseline of 74.1% in 2016 to 79.6% in December 2017, short of the original target of 90% and exceeding the revised target of 70%.  The network of programs that were using Cadastro Único reached 29 programs. Page 6 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Bolsa Familia 2nd APL (P101504)  The project developed a web portal allowing citizens to verify the information content about themselves in the Cadastro Único social registry.  The inclusion of indigenous groups, Quilombolas, and the homeless increased from 200,000 persons in 2011 to 1.9 million people in 2015.  An inventory of Bolsa Familia evaluations was developed and included 1,070 publications, as well as a book titled “15 anos do Bolsa Família” or “15 Years of Bolsa Familia” that provided a summary of 190 peer-reviewed studies about the impact of BFP. Outcomes The percentage of households in the first quintile receiving transfers from BFP increased from a baseline of 66% in 2011 to 68.1% in December 2016, short of the original target of 75%, but almost achieving the revised target of 70% (the revised target considered methodological issues in data collection and comparability, as noted in Section 2a). Between 2004 and 2015, extreme poverty in Brazil decreased from 7% to 3%, and overall poverty from 21% to 9%. Concerning attribution, it is understood that poverty reduction is multi-factorial, but it would be reasonable to assume that the project was among the contributing factors. Estimates made in 2016 suggested that social transfers accounted for 58% of the decline in extreme poverty, i.e., benefiting four million people who were previously in extreme poverty, and for 30% of the decline in poverty, equivalent to 8.1 million people (ICR, p. 12). The Gini coefficient that gauges economic inequality dropped from 0.59 in the previous decade to 0.52 in 2015. As above, it would be reasonable to assume that BFP was a contributing factor, among others, to this somewhat favorable pattern. The number of state governments using the single registry for the selection of beneficiaries for their state social programs increased from a baseline of 9 states in 2011 to 24 states in December 2017, exceeding the target of 21 states. Rating Substantial OBJECTIVE 2 Objective Page 7 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Bolsa Familia 2nd APL (P101504) Strengthen the ability of Bolsa Familia Program to promote human capital development. Rationale Operationally, BFF sought to promote human capital in terms of improved education and health through conditionalities and enhanced monitoring of such conditionalities with the education and health sectors that provided the actual services in education and health. It was reasonable to expect that, in addition to program- strengthening outputs described above under Objective 1, facilitating access and use of social, educational, health, and nutrition services, house visits for basic health care delivery, and care under the early childhood development program, reaching children and pregnant women, and improving the monitoring of related conditionalities would contribute to better education and health results, which in turn would contribute to human capital development. Outputs and intermediate results  In addition to technical assistance and training, the project monitored conditionalities related to education and health. 13.4 million students were monitored, and 94.6% met education co- responsibilities. 5.4 million children were monitored in health, of whom 84% percent had regular growth monitoring, while 99% percent complied with vaccination responsibilities.  Under the project, policy guidelines were established for early childhood development, and a qualitative study to evaluate the program was prepared. 13,078 visitors and 3,270 supervisors were trained for program protocols, activities, and home visits. The program reached 351,640 children age 0-3 years and 64,296 pregnant women in 3,780 municipalities.  The number of agreements signed with the states and implemented increased from a baseline of zero in 2012 to 27 in 2017, exceeding the original target of 13, and achieving the revised target of 27. Outcomes The percentage of youth (16–17 years old) whose school attendance was monitored by the program increased from a baseline of 75.9% in 2011 to 84.8% in 2017, virtually attaining the target of 85%. The percentage of beneficiary children of primary school age (6–15 years old) whose school attendance was monitored by the program increased from a baseline of 88.1% in 2011 to 94.9% in 2017, exceeding the target of 90%. Page 8 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Bolsa Familia 2nd APL (P101504) The percentage of children aged 0–6 and pregnant women in beneficiary families complying with health conditionalities increased from a baseline of 70% in 2011 to 75.2% in 2017, attaining the target of 75%. Comparative studies showed favorable results related to human capital development. Municipalities with high BFP program coverage had an average of 19% lower under-five mortality than similar municipalities with low BFP coverage; and municipalities with high program coverage had 53% lower under-five mortality due to diarrhea, and 65% lower under-five mortality due to malnutrition, than low coverage municipalities (ICR, p. 18). The ICR noted that it was plausible that output enhancements made through the program, including vaccinations, breastfeeding, access to food, and overall support and care, contributed to these outcomes, further facilitated by the fact that 88% of primary recipients of cash transfers were women, and, to the extent possible, payments were made directly to the mothers of beneficiary children. Similarly, municipalities with the highest monitoring levels of school attendance had the highest school progression and lowest dropout rates, indicating positive effects of BFP conditionalities. Also, per TTL clarifications (10/29/19), an increase of 33% in high school completion was reported in 2015 among program beneficiaries. Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The aggregation of two almost fully achieved objectives indicates a Substantial rating for overall efficacy. Overall Efficacy Rating Substantial 5. Efficiency At appraisal, the project’s economic analysis did not undertake a cost-benefit analysis or cost-effectiveness analysis. It presented the results of an ex-ante evaluation on the role of Bolsa Família in mitigating the effects of the food price increase in 2008 (PAD, p. 22), where it was suggested that average changes in food prices reduced per capita income by 2%, and maximum food price changes reduced average per capita income by 4.6%. It concluded that cash transfers could buffer the impact of food prices on the most vulnerable. Page 9 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Bolsa Familia 2nd APL (P101504) The ICR did not undertake a cost-benefit or cost-effectiveness analysis. The ICR noted improvements in education and health indicators, many of which were referenced in the published "Bolsa Família 15 Anos or BFP 15 Years,” such as improved school attendance of children in beneficiary households, reduction in repetition, and favorable trends in health service utilization, fertility, and food consumption. Hence, the ICR's economic analysis highlighted gains toward which the project has contributed, rather than addressing efficiency aspects of the project itself. Such gains are more suitably considered under efficacy. For example, in education, improvements in school attendance, progression, and dropout rates may have resulted from the project, suggesting improved internal efficiency, but they do not necessarily indicate that project resources were used efficiently. Significant shortcomings in the efficiency of implementation were noted by the ICR, which described the efficiency of project implementation as "not positive" (ICR, p. 24). Project effectiveness had a considerable delay of 14 months between approval on 9/16/10 and effectiveness on 11/23/11. The delay was explained by a change in presidential administration and policy developments in social assistance and poverty reduction by the new administration (ICR, p. 27). In 2011, the new government launched a new plan, PBSM, whose objective was to increase capacities and opportunities for the extreme poor. The government restructured MDS and created a new secretariat to coordinate new program activities. The new priorities along with the institutional changes in MDS resulted in the project’s first restructuring in 2014 (ICR, p. 28). In 2016, another MDS restructuring created two new national secretariats. Several high-level and experienced technical staff left MDS, and replacements were delayed, leading to a loss of technical capacity and institutional memory (ICR, p. 28). During this transition, several Bank implementation support missions were delayed at the client’s request until new staff were in place. According to the ICR (p. 29), the project was implemented in a very difficult macroeconomic environment. The ICR stated that after 2013, the economy stumbled and fell into a deep recession. Fiscal consolidation and austerity affected federal expenditures, and subnational governments faced a solvency crisis due to high rigidities in their expenditures. This led to staffing constraints under the project. The long political transition, particularly in 2016–2017, brought technical assistance disbursement to a standstill. The economic crisis brought a sharp depreciation of the Brazilian real, which led to a further reduction in drawdowns from the designated account. The ICR noted that project implementation was affected by political turnover, and that the uncertain political and economic situation affected the ability of technical secretariats to engage in medium-term policy and expenditure planning or to execute previously planned project activities. Intended technical assistance activities were negatively affected by constraints imposed on planned expenditures in all government branches (ICR, p. 29). The ICR noted improvement in technical assistance disbursement after 2017, although it also noted that a government-wide administrative expenditure cap in 2018 affected the amount of resources that MDS could allocate to consultancies, regardless of their financing source (ICR, p. 24). At the MDS Project implementation Unit, procurement delays impacted implementation (ICR, pp. 29-31), with key steps falling behind established dates in yearly procurement plans. There were issues of noncompliance by consultants, resulting in the cancellation of awarded contracts. The ICR noted that there was a significant slowdown in technical assistance disbursements after 2016. Newly contracted teams had to be familiarized with procurement processes. The high numbers of individual consultancy procurements were explained by MDS’s Page 10 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Bolsa Familia 2nd APL (P101504) risk aversion to hire larger consultancies for field work because of past negative experience. The project had two extensions aggregating at three years, thus extending the project period to eight years. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of objectives is rated High, as the objectives remained fully aligned at project closing with the Bank’s Country Partnership Framework for the Federative Republic of Brazil, FY18– FY23, and with the country’s national plans. Efficacy is rated Substantial, as the project almost fully achieved its development objectives. Efficiency is rated Modest because of deficiencies in economic analysis and significant shortcomings in the efficiency of implementation. Therefore, the overall outcome is rated Moderately Satisfactory, indicative of essentially moderate shortcomings in the project’s overall preparation, implementation, and achievement. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome There are no noteworthy technical risks or institutional risks, but rather economic and fiscal risks. CPF FY18– FY23 stated that wavering political support for pension reform would leave little fiscal space to expand safety net programs like Bolsa Família to ensure the protection of the most vulnerable and the new poor. However, the government has so far maintained a program allocation equivalent to US$8.1 billion. The institutional framework and national capacities in Brazil are strong, and MDS merged with the Ministries of Culture and Sports to become the new Ministry of Citizenship. In the short term, continuity in BFP and complementary programs appears likely, given that the previous minister was designated to lead the new ministry, and with sustained overall public sector commitment to this flagship program. At the same time, the Page 11 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Bolsa Familia 2nd APL (P101504) ICR (p. 32) noted the preference of some policy circles, outside MDS, to have lower program coverage, reflecting concerns about financial sustainability. 8. Assessment of Bank Performance a. Quality-at-Entry As the project constituted the second APL phase, preparation built on an existing base provided by the previous phase. Preparation also built upon well-established relations of partnership between the government and the Bank, enhanced by a shared interest in sustaining program results with further program strengthening. Challenges were jointly identified, and preparation incorporated lessons learned during the first operation as well as findings of the AAA program (Analytical and Advisory Assistance, a term later changed to Advisory Services and Analytics or ASA) between the Bank and Brazil. Lessons reflected in the project design included the need for clarity in institutional responsibilities, transparency of rules, transparency of targeting mechanisms, and strengthening human capital, notably in education, health, and nutrition (PAD, pp. 10-11). The PAD did not undertake a cost-benefit analysis or cost- effectiveness analysis (see Section 5), but referred to the effectiveness of CCTs in improving human capital in the literature. Institutional, fiduciary, and implementation arrangements were adequately prepared and built on the same arrangements used under APL1. MDS was the main implementation agency. While the co-financed cash transfers were to be directly disbursed to the Ministry of Finance, technical assistance components were managed by MDS, with an Executive Secretariat and several technical secretariats responsible for the technical aspects of project activities (PAD, p. 13). Most risks were adequately identified and mitigation measures were planned (pp. 16-20), but macro-economic risks were not considered, and the PAD stated that “no major fiscal constraints were foreseen within the next five years, despite the international financial and economic crisis” (PAD, p. 16). However, related mitigation measures would have been challenging, as they would extend beyond the project. Quality-at-Entry Rating Satisfactory b. Quality of supervision The project team reportedly provided regular supervision and implementation support, and adequately monitored and addressed fiduciary and safeguard aspects (ICR, p. 32). The team provided quality reporting, and Implementation Status and Results Reports were timely and thorough. The project team navigated effectively in an environment of economic crisis and political difficulties, as there were two changes in government administrations marked by a presidential impeachment and a subsequent electoral Page 12 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Bolsa Familia 2nd APL (P101504) period. Political changes were associated with shifts in priorities and institutional modifications at the implementing agency. The Bank team demonstrated pro-activeness, including through two project restructurings, to address arising issues and to facilitate the pursuit of intended results through adaptation to new structures and activities. Both the team and government counterparts maintained their primary focus on strengthening BFP ability to achieve its stated objectives. This ICR Review notes that shortcomings in the efficiency of implementation were beyond the control of the Bank team, and in fact, the team appeared to have made superlative efforts to mitigate the unexpected political and economic challenges. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The project provided broadly stated objectives. There was a shortcoming in the conceptualization of the theory of change specific to the project development objective as stated (see Section 4): the PDO was stated in terms of a change in institutional capacity (to “strengthen BFP’s ability to achieve the objective of reducing poverty and inequality and promoting human capital development…”), while the ICR’s discussion of the theory of change seemed to assume the objective was actually “BFP’s impact on poverty and inequality reduction, and human capital development.” Also, reduction in poverty and improvements in human capital are ordinarily understood as arising from a combination of many social and economic factors, beyond the program. Hence, the project design could have elucidated the limited contributory role of the program. Nevertheless, the indicators were generally relevant to the objectives, and the results chain established the links between key activities, outputs, and outcomes reflected in the results framework. M&E arrangements built on well-developed monitoring systems at MDS and its Secretariat for Information Management and Evaluation. Baselines were available. M&E functions and processes were likely to be sustained after project closing since they built on existing and adequately functioning country systems. Three impact evaluation rounds were also planned. b. M&E Implementation According to the ICR, the quality of data was generally high, although there were methodological and comparability issues among surveys that measured program coverage for the poorest quintile (ICR, pp. 12-13). The project regularly updated progress on indicators and forwarded data to the Bank every six months. The two project restructurings revised intermediate results indicators and introduced new ones, Page 13 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Bolsa Familia 2nd APL (P101504) including a downward revision in 2017 for one outcome indicator (percentage of households in the first quintile receiving BFP transfers) from 75% to 70%, because of methodological issues noted above. The project also introduced a new module to the National Household Survey to analyze access and results of productive inclusion programs and coverage of Cadastro Único. The third round of the impact evaluation or Poverty Panel Survey was not carried out due to multiple reasons, including compliance issues by the contracting firms, and quality issues with baselines (TTL clarifications, 10/21/19). c. M&E Utilization In addition to regular utilization of M&E for project operations and monitoring, the project entered into agreements with 29 government programs, and programs of 24 federal units, to utilize the information of Cadastro Único (ICR, p. 30). MDS made data available on its external website, thereby enabling data use for dialogue with stakeholders while also catalyzing further studies by external groups and global academia. According to the ICR (p. 19), MDS shared BFP evaluations through an online database, including 1,070 publications, 108 of which were carried out by international scholars, but it did not provide a link to the database. The report “15 Years of Bolsa Familia” provided a summary of 190 studies in peer-reviewed journals about the impact of BFP (Bolsa Família 15 Anos, 2003- 2018: https://repositorio.enap.gov.br/handle/1/3647). The ICR (p. 26) stated that BFP was considered to be a model of a well-functioning and targeted cash transfer program, and that there was considerable interest in other countries to learn from Brazil’s experience. In March 2013, a memorandum of understanding was signed between MDS, the Bank, and national and international agencies to establish a “Knowledge and Innovation Initiative on Poverty Reduction," including for sharing knowledge with other countries in the region and globally. The” World Without Poverty," an inter-institutional partnership, was established in 2013 between the International Policy Center for Inclusive Growth, Brazil’s Institute for Applied Economic Research, MDS, and the Bank to share Brazil’s innovative social protection policies and to help other countries deepen their knowledge about BFP and the Cadastro Único. Between 2014 and 2017, about 80 products on BFP implementation were made available online for international exchange. M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project did not trigger environmental policies and was classified as Environmental Assessment Category C, but it triggered safeguard policy “OP/BP 4.10 Indigenous Peoples” in view of the project’s inclusion of indigenous populations and Quilombolas (traditional communities) among BFP beneficiaries. The BFP framework for differentiated attention to indigenous people was consistent with the safeguard policy. There were 658 Social Assistance Reference Centers providing services to indigenous people and 973 Centers to Quilombolas, including outreach teams to assist remote communities. The ICR Page 14 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Bolsa Familia 2nd APL (P101504) noted that, by 2018, 151,800 indigenous families had been registered under the Cadastro Único. Bank missions consistently reported full compliance with safeguard policies (ICR, p. 31). b. Fiduciary Compliance Financial management. The project’s financial management arrangements were adequate and complied with national and Bank procedures, including for procurement. Project accounts were audited annually, and the audits were unqualified throughout implementation (ICR, p. 31). Procurement was undertaken by MDS’s project implementation unit and by a procurement agency for non-consulting services. The main procurement issues that affected implementation were related to delays in procurement processes (see Section 5) and noncompliance by consultants that resulted in the cancellation of awarded contracts. The ICR also noted (p. 31) that migration of procurement processes to the Bank’s STEP system presented challenges during the final stages of implementation. c. Unintended impacts (Positive or Negative) According to the ICR (p. 26), and as stated in Section 9c, a memorandum of understanding was signed in March 2013 between MDS, the Bank, and national and international agencies to establish a “Knowledge and Innovation Initiative on Poverty Reduction," to assist in overcoming extreme poverty and promoting more inclusive development, and in sharing knowledge with other countries in the region and globally. Also, the “World Without Poverty," an interinstitutional partnership, was established in 2013 between the International Policy Centre for Inclusive Growth, Brazil’s Institute for Applied Economic Research, MDS, and the Bank to share Brazil’s innovative social protection policies and to help other countries deepen their knowledge about BFP and the Cadastro Único. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment The ICR rated efficiency as Outcome Satisfactory Moderately Satisfactory Substantial, and this ICR Review rated efficiency as Modest Page 15 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Bolsa Familia 2nd APL (P101504) because of deficiencies in economic analysis and significant shortcomings in the efficiency of implementation that were beyond the control of the Bank's implementation support team. Bank Performance Satisfactory Satisfactory Quality of M&E Substantial Substantial Quality of ICR --- Modest 12. Lessons The ICR (pp. 33-34) offered several lessons and recommendations, including the following lessons restated by IEG: A comprehensive and regularly updated beneficiary registry facilitates the implementation not only of a cash transfer program, but also its use by other targeted programs. Under the project, BFP’s single registry of beneficiaries was used by 29 programs. The development and disclosure of a large body of evidence on a large conditional cash transfer program enhances credibility and stimulates further knowledge and studies by external groups at no cost to the government. Under the project, MDS made data available on its external website, thereby promoting transparency and enabling the use of rich data for further studies undertaken by external groups. Initial positive results generated by a social protection program help in building further support and increasing prospects for sustainability. Despite political turnover, commitment to BFP remained strong, as the program was perceived as an effective scheme in confronting poverty, as documented by its early results. 13. Assessment Recommended? No 14. Comments on Quality of ICR Page 16 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR Bolsa Familia 2nd APL (P101504) The ICR was well organized and results-oriented. The theory of change was reasonable and generally aligned to development objectives, although improvements in capacity and improved outcomes were assumed to be interchangeable. The evidence was generally of good quality and built on findings within and beyond the project. The discussion of M&E was thorough. The ICR’s narrative and conclusions drawn are supported by the evidence provided, except for project efficiency. The ICR was generally consistent with guidelines. Lessons were directly derived from project experience. However, the ICR had a notable shortcoming, as it did not undertake a cost-benefit or cost-effectiveness analysis. a. Quality of ICR Rating Modest Page 17 of 17