ICRR 12543 Report Number : ICRR12543 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 11/27/2006 PROJ ID :P057910 Appraisal Actual Project Name :Br Pension Reform Lil Project Costs 10.0 9.0 US$M ) (US$M) Country :Brazil Loan/ US$M ) Loan /Credit (US$M) 5.0 4.5 Sector (s):Board: ): PS - Compulsory US$M ) Cofinancing (US$M) 0.0 0.0 pension and unemployment insurance (100%) L/C Number :L4369 FY ) Board Approval (FY) 98 Partners involved : None Closing Date 12/31/2001 12/01/2005 Evaluator : Panel Reviewer : Division Manager : Division : Jaime Jaramillo Roy Gilbert Lily L. Chu IEGCR 2. Project Objectives and Components a. Objectives The objectives defined in the 1998 PAD were: To prepare the majority of the Brazilian states technically to address pension reform and help them move through the initial phases of reform, that is the development of the diagnostic tools and the reform proposals; and To assist the federal and state governments to choose among the best options and encourage them to implement sound reform. b. Components (or Key Conditions in the case of Adjustment Loans ): Diagnostic Tools : To provide for 15 of 26 states the tools to understand their situations and develop their reform proposals, including: Development of a database with a clean sample of 15 percent of the active and retired employees in participating states; Training of two officials per state in pension reform and the use of the Bank's Pension Reform Options Simulation Toolkit (PROST); and Development of a Technical Unit (TU) within the Federal Pension Ministry (MPAS, later MPS) with the legal mandate and resources to assist the states and the federal government on pension reform . Reform Proposals : To develop pension reform proposals in the states with the diagnostic tools . The preparation included disseminating the proposals and debating in open fora . Institutional Framework : To move forward with the legislative and institutional frameworks needed for enacting the reforms in the states that developed the reform proposals . Government Strengthening : Through: Drafting of complementary legislation to implement the Constitutional Amendment of 1998; Sharing best practices for pension fund management; Designing a mechanism for federal pension related obligations to move to the states and state obligations to the National Social Security Institute (INSS); and Sharing experience of different countries with pension reform and identification of the steps needed to ensure the consistency of state reforms with the reform of the federal social security system . c. Comments on Project Cost, Financing, Borrower Contribution, and Dates The project became effective a year after approval due to the financial crisis faced by Brazil in 1998. Management extended this LIL three times : in December 2001, November 2003, and July 2004. Furthermore, in February 2004 Brazil cancelled US$500,000 from the loan amount, reducing the total amount of the loan to US$ 4.5 million. The ICR reports that total project costs were about US$ 9 million, half of which financed by the Bank, but without a breakdown--this may not be accurate, given other errors in the tables . 3. Relevance of Objectives & Design : The project objectives were highly relevant, in terms of its consistency with both Brazil's priorities and the Bank's strategy as defined in the successive CASs (1997, 2000, and 2003). It also helped put in motion a pension management system that reduced redundant expenditures in pensions . The project was approved at a time when Brazil was facing a severe financial crisis, and the federal government was pushing a constitutional amendment to deal with pension issues, which was finally approved in December 1998. The amendment sought, among other things, to bring actuarial and fiscal balance to the pension systems, limit special pension regimes and benefits (there were nearly 2200 regimes), and provide some leeway for state and municipal governments to address the pension issues. The presidential elections of 2002 brought in a new team with a different perspective on pension and social issues, but it wholeheartedly supported the project once it became acquainted with it . The initial project design was not relevant because it did not factor in key elements, but the design gained relevance as it was adapted using the LIL's flexibility . The key elements missed initially included the following : The project intended to use a sample to estimate the savings that could come from pension reform . Brazil, instead, wanted and opted for preparing a registry of all the employees and pensioners ("cadastro" in Portuguese). The complete registries allowed the weeding of the pension bills, generating significant fiscal savings and creating the basis for a sound management of the pension systems . The project included the use of the Bank's PROST software . Brazil found PROST inadequate, and developed instead its own software (SIPREV), which was better suited to carry out the actuarial assessments with the complete registries and facilitated the planing and administration of the participating state pension systems . The project assumed that Brazil's legal system would allow straightforward pension reform triggered by the educated guesstimates of PROST, and that the real issue was making a sound choice regarding the system . In reality, the constitutional amendment of 1998 was not enough to provide this flexibility, and the system remains rigid even after the constitutional amendment of 2003--the Constitution still gives the states very little leeway to define their pension systems . The PAD did not acknowledge the existence of almost 2200 different pension regimes in Brazil, and the legal complexity that this issue entails . The team and the Bank showed a commendable flexibility in adjusting to circumstances and developments, using the LIL framework to continue the dialogue with Brazil and provide support throughout the different extensions of the project. 4. Achievement of Objectives (Efficacy) : Preparing Brazilian states technically to address pension reform and help them move through the initial phases of reform: High. The project helped improve pension management in 15 states, reportedly yielding potential annual savings of about US$ 255 million in 9 states (the ICR does not provide information on the remaining entities). The improvement was achieved by developing two technical elements and a management unit that are crucial for this purpose. One technical element was the preparation of registries of all employees and pensioners (as opposed to a sample of 15 percent envisioned initially) for each entity, which helped stop the leaks due to unlawful pension payments . The other one is the development by the Brazilians themselves of their special software for comprehensive pension management (SIPREV). The project also fostered the development of a management unit in MPS that generated a new model of sub -national collaboration, as the project's technical unit gradually assumed this role . The borrower acknowledges that, through this management unit, the project created "a natural network for the dissemination and replication of results among these entities, " that was instrumental in bringing about the better than expected coverage of states and the movement forward in pension reform. Assisting the federal and state governments to choose among the best options and encourage them to implement sound reform : Modest. The project allowed the states and the federal government, who faced severe legal constraints, to opt for focusing mostly on improving pension management --this improvement was what brought about the significant fiscal savings . The legal and constitutional reforms that took place were undertaken by the Federal Government and, in addition to this project, the Bank was also engaging Brazil in substantive policy dialogue on pension reform through two sectoral adjustment loans (1999 and 2000). Both the IMF and the IADB were engaged as well with Brazil in policy dialogue on pension reform . 5. Efficiency : With the limited information available in the ICR, it would appear that the project was very efficient . The ICR provides information on the potential annual savings of only 9 states, and on the costs of the project that were financed with the loan from the Bank. Even assuming that there were no savings in the remaining entities and that total costs were 10 times higher than the Bank's loan, the FRR would be about 250 percent. The ICR, however, does not provide any information on these remaining entities, and reports total costs, including developing the registries, software, training of over 1000 entities and hardware, of only US$ 9 million. Timewise, however, the project extensions suggest a lengthy process that perhaps may have been shortened if the project's design had been more relevant . 6. M&E Design, Implementation, & Utilization: According to the Project Design Summary of the PAD, the Bank's team assumed only a limited number of M&E activities, including on the completion of the Diagnostic Tools in 15 states, and the establishment and training of technical units in participating states and the federal government . The staff made eight visits during the seven years of life of the project. The ICR, however, provides very skimpy information on what came out of those visits and the use of that information, noting that "evaluators could not find a complete set of early BTORs and Project Status reports in the files." 7. Other (Safeguards, Fiduciary, Unintended Impacts--Positive & Negative): None. 8. Ratings : ICR ICR Review Reason for Disagreement /Comments Outcome : Highly Satisfactory Satisfactory The registries, software and management unit have led to a pension management system that appears to yield very significant fiscal savings. However, achievement of the second objective was only modest. Institutional Dev .: High High Sustainability : Highly Likely Highly Likely Bank Performance : Satisfactory Satisfactory Flexibility in implementation made up for the shortcomings in project design, which could have been better grounded on the actual conditions. Borrower Perf .: Satisfactory Highly Satisfactory The Government of Brazil was the key force behind the success of the program and the large fiscal savings, because it led the way by: Expanding the project by preparing the full registries of employees and pensioners; Developing its own software tailored to its own issues; and Creating a sub-national collaboration model and a pension management system that triggered large fiscal savings, even after the change in administration and the financial crisis. Quality of ICR : Unsatisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - ICR rating values flagged with ' * ' don't comply with OP/BP 13.55, but are listed for completeness . 9. Lessons: Keeping an eye on results and flexibility in achieving them seems to be a more effective and preferable approach than the traditional one of focusing only on a specific policy prescription . The flexible approach taken by the team played a major role in the success of the project . Indeed, the team accepted many changes--registries, software, management system, while remaining focused on helping Brazil improve a state pension system that was perceived by all as a fiscal threat . The Bank needs to have a better grip of existing conditions on the ground before embarking on a project . Projects prepared in the midst of critical periods may fail to factor in the constraints and opportunities faced by the borrower. In this particular case, the financial crisis of 1998 may have played a role in the initial lack of relevance of the design of the project . Perhaps a better designed project could have yielded the fiscal savings in a shorter period. The Bank also needs to better understand the potential and limitations of its technical instruments in projects such as this one. Better project preparation may have saved a lot of effort and aggravation on issues such as the adequacy of PROST for the task in hand, the usefulness of 15 percent samples, and how much reform was feasible within the existing constitutional and legal constraints . Institutional development may be preferable to legal reform in cases such as this one . The success of the project in terms of fiscal savings appears to have came about with the development of a pension monitoring and administration system. From the available information, it is difficult to imagine how the outcome could have been better had Brazil implemented the legal reforms that the PAD had initially deemed necessary . Understanding the true benefits of a project and their attribution requires information on the actual and potential benefits stemming from it, as well as on the total costs associated with it . In this particular case, the team could have kept track of the actual fiscal savings attributable to the project, as well as of the entire costs incurred by Brazil in implementing it. The team could have also have kept track of the institutional and regulatory changes that were happening because of the project . In the absence of such information it is difficult to know who deserves the credit for its success . 10. Assessment Recommended? Yes No Why? The Bank's dialogue with Brazil on pension reform is reported to have yielded substantive results . The dialogue took place through several channels, including two sectoral adjustment loans, this LIL and several pieces of ESW. In addition, the IMF and the IADB were also active in this area of dialogue at about the same time . An assessment would help elucidate the issue of attribution in a credible way . It would also help distill the lessons of success of the different channels used by the Bank, which could be useful for future teams working with Brazil and for teams working with countries with similar circumstances as Brazil . 11. Comments on Quality of ICR: The ICR failed to provide the necessary information on the benefits, costs and extensions of the project, while exaggerating the Bank's contribution in topics such as the 2003 constitutional amendment. Indeed, Brazil's comments in the ICR, while acknowledging the Bank's help in developing the software and registries, fail to back the ICR's claim about the importance of the Bank's support in the ensuing constitutional and legal reforms . In several places the ICR recognizes the lack of information, yet it goes on to draw conclusions on the basis of no evidence . For the most part, the ICR reads like a recent history of pension reform in Brazil as a whole, without attempting to attribute particular results to the project itself, nor acknowledging the contributions of Brazil and other partners such as the IMF and the IADB. Moreover, the ICR incorrectly inserts a third objective, ex -post, into the project ("to build a constituency for reform"). The ICR also provides a list of eight "loan objectives"--most of which are not objectives at all--that do not correspond to the declared aim of the project . As well as being inconsistent with the project's intent, the ICR fails to recognize the limited scope of objectives of LILs . As noted above, the ICR's statistical information, especially on costs, lacks precision, consistency and, possibly, completeness . Somewhat surprising is the complete lack of information of the actual fiscal savings attributable to the project in the 15 states that adopted the registries and the software, as well as the expected potential savings --reporting potential savings for a third of the states and for what was expected one full year before the ICR is pointless . The ICR also contains mistakes on the Project Data section, such as the inclusion of David De Ferranti as LAC VP at the time of appraisal .