The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20 An Independent Evaluation © 2022 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW Washington, DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org ATTRIBUTION Please cite the report as: World Bank. 2022. The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20. Independent Evaluation Group. Washington, DC: World Bank. COVER PHOTO Adapted from shutterstock/ Sittipong Phokawattana EDITING AND PRODUCTION Amanda O’Brien GRAPHIC DESIGN Luísa Ulhoa This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The bound- aries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. RIGHTS AND PERMISSIONS The material in this work is subject to copyright. Because The World Bank encourages dissem- ination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Any queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20 An Independent Evaluation March 15, 2022 Contents Abbreviations v Acknowledgments vi Cover Note viii Overview xi Management Response xxi Report to the Board from the Committee on Development Effectiveness xxiii 1. Introduction: Doing Business and Country Reforms������������������������������������������������ 1 Background and Context 2 Influence 3 Controversy 7 Evaluation Approach 9 The Relevance of Doing Business: Is It “Doing the Right Things”?��������������������������11 2.  Client Country Reforms and Development Goals 13 Business Area Relevance of Doing Business Indicators 23 World Bank Group Country Strategy and Operations 30 Research 35 Doing Business Indicators over Time 36 The Effectiveness of Doing Business: Is It “Doing Things Right”?��������������������������39 3.  Research Evidence 41 Effectiveness Claims Made in Doing Business Reports 44 Country Reforms 47 World Bank Group Country Strategies 52 Learning from Factors of Project Success 57 Recommendations to Improve the Use of Doing Business in Country Reforms��63 4.  Bibliography��������������������������������������������������������������������������������������������������������������� 69 ii Boxes Box 1.1. What Are the Doing Business Indicators? 4 Box 2.1. Views of World Bank Group Expert Practitioners 12 Box 2.2. Doing Business Reform Memorandums 16 The Perception of World Bank Group Conflict of Interest Box 2.3.  between Indicator Generation and Operational Work 17 Box 2.4. Doing Business Reform: Traction in China 18 Colombia: Broadening the Evidence Base for Reform, Fiscal Years 2010–20 20 Box 2.5.  Box 2.6. Caveats in Doing Business Reports, 2010–20 27 Four Doing Business Outcome Findings Evidenced Box 3.1.  by Multiple Articles Published in Leading Journals 42 Comparing the Doing Business 2020 Literature Review with the Box 3.2.  Independent Evaluation Group Structured Literature Review 46 Figures Figure O.1. Doing Business Indicator Expansion over Time ix How Doing Business Influences Actions Leading to Figure 1.1.  Development Outcomes 6 Change title to: Positive National Reforms Tracked by DB2010–DB2020 Figure 2.1.  14 Correlation between the Doing Business Ease of Doing Business Overall Figure 2.2.  Score and the World Economic Forum Burden of Regulation Indicator 24 Indicators Used as Measures of Business Environment Figure 2.3.  Constraints in Country Private Sector Diagnostics 31 Share of Doing Business–Informed Projects in World Bank Figure 2.4.  Group Portfolio, Fiscal Years 2010–20 32 Projected Doing Business–Informed Portfolio FY10–20 Figure 2.5.  by Indicator Area and World Bank Group Institution (number and percent of DB-informed projects) 34 Rigorous Studies of Outcomes Identified by Structured Literature Review Figure 3.1.  Relating to Indexed Doing Business Indicators 43 iii Claims Linking Doing Business–Related Reforms to Figure 3.2.  Development Impacts 46 Project Success by Doing Business Indicator Area, Fiscal Years 2010–20 Figure 3.3.  56 Success Rate of Doing Business–Related Intermediate Figure 3.4.  Outcomes, Fiscal Years 2010–20 57 Factors Influencing Outcomes in IEG-Evaluated Projects, Fiscal Years Figure 3.5.  2010–20 59 Tables Table 1.1. Findings of Past Evaluations and Reviews of Doing Business 7 Correlations and Over- or Underestimating DB versus Enterprise Surveys 25 Table 2.1.  Key Findings on Relevance from Doing Business Indicator Deep Dives 28 Table 2.2.  Summary of Doing Business–Informed Portfolio, Approved in Table 2.3.  Fiscal Years 2010–20 (projection) 32 Key Findings on Effectiveness from Doing Business Indicator Deep Dives Table 3.1.  49 Success Rate of Doing Business–Related Interventions Table 3.2.  by Type of Use of the DB Report, Fiscal Years 2010–20 56 Lessons from World Bank Lending and International Table 3.3.  Finance Corporation Advisory Services Projects 60 Appendixes Appendix A. Methodology 78 Appendix B. Portfolio Review 100 Appendix C. Case Studies 121 Appendix D. Indicator Area Deep Dives 157 Appendix E. Desk Review of Literature 184 Appendix F. Structured Literature Review (SLR) 200  ummary of the Analysis of WBG Country Appendix G. S Strategies and Frameworks 232 Appendix H. Summary of Econometrical Findings 242  ummary of Findings related to IFC AS Appendix I. S and WB Lending Evaluations 263 Appendix J. Review of Claims and their References 284 iv Abbreviations AS advisory services CPE Country Program Evaluation CPSD Country Private Sector Diagnostic DB Doing Business DEC Development Economics Vice Presidency DWCP dealing with construction permits EoDB ease of doing business FY fiscal year IEG Independent Evaluation Group IFC International Finance Corporation M&E monitoring and evaluation SLR structured literature review SME small and medium enterprise NCITRAL United Nations Commission on International Trade Law U All dollar amounts are US dollars unless otherwise indicated. Independent Evaluation Group World Bank Group    v Acknowledgments Andrew Stone, lead evaluation specialist, led this evaluation for the Inde- pendent Evaluation Group (IEG) Financial, Private Sector, and Sustainable Development Department, under the guidance of José Candido Carbajo Mar- tinez, director, IEG Financial, Private Sector, and Sustainable Development, and Alison Evans, director-general of IEG. Nadia Paola Ramirez Abarca led the portfolio analysis and coordinated the The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Acknowledgments analyst team, which included Ayooluwa Olusola Adewole, Mariana Calderon Cerbon, Regina Legarreta, Victor Malca, Anna Mortara, Ozlem Onerci, Na- thaniel Russell, and Rasec Niembro Urista. Case study authors included Amitava Banerjee, Andrew Stone, Anjali Kumar, Anna Mortara, Ayooluwa Adewole, Jingwen Zhen, Nadia Ramirez, Paul Holden, Rasec Urista, Regina Legarreta, and Riad al Khouri, with Emily Harwitt and Mujtaba Basij Rasikh. Deep dives studies were authored by Ayooluwa Adewole (getting credit), Paul Holden (starting a business), Regina Legarreta and Anna Mortara (paying taxes), Andrew Stone (trading across borders), and Rasec Urista (dealing with construction permits), as well as Ana Belen Barbeito (International Finance Corporation advisory services) and Melvin Vaz (Project Performance Assessment Reports). Giuseppe Iarossi (with Silvia Espinosa) and Anqing Shi authored statistical and econometric analytic inputs. Paul Fenton-Villar authored the structured literature review. Harsh Anuj worked with the team to produce the supervised machine learning inputs, and Ariya Hagh provided quality control on the econometrics. Emelda Cudilla provided administrative support and formatted the report. Helpful comments and guidance were received from external peer review- ers: Dean Judith Kelley, Vijaya Ramachandran, and Vincent Palmade, with Alvaro Gonzalez. Further guidance was provided by an ad hoc advisory panel composed of Jaime Roberto Diaz, Jeffrey Chelsky, Marialisa Motta, and Peter Lanjouw. Jozef Leonardus Vaessen advised the team on methods. The team thanks key counterparts in the World Bank and the International Finance Corporation. We particularly thank Deepa Chakrapani, Aart Kraay, Zhenwei Qiang, Martin Raiser, Rita Ramalho, and Sylvie Solf for important interac- vi tions during this evaluation’s preparation, as well as the 10 country teams and offices that facilitated virtual missions. However, IEG bears sole respon- sibility for the content of this evaluation. Independent Evaluation Group World Bank Group    vii Cover Note This note details the chronology of steps in this evaluation’s production before the decision by the World Bank Group’s management on September 16, 2021, to discontinue its Doing Business (DB) flagship report. It also presents a sum- mary of the lessons drawn from the evaluation that are relevant to the design of any future approach to assessing the business and investment climate. Chronology The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Cover Note In June 2020, the Committee on Development Effectiveness of the Board of Executive Directors the Bank Group asked the Independent Evaluation Group (IEG) to evaluate the DB indicators, including the Ease of Doing Business country rankings, to “assess the way that DB aligns with the Bank Group’s mandate and a viable development framework for client countries.” In September 2020, IEG produced an Approach Paper defining a focus on the assessment of DB’s strategic relevance and effectiveness for client coun- tries’ reform priorities and to the Bank Group’s strategic agenda using IEG’s mandated lens of development effectiveness. It acknowledged parallel work being undertaken by the Bank Group’s Group Internal Audit and Develop- ment Economics Vice Presidency. The three bodies consulted and exchanged information to assure complementarity. In March 2021, IEG began producing an Issues Paper for internal and man- agement review that was submitted to the Committee on Development Effectiveness on June 29, 2021. It identified six lines of inquiry for the main DB evaluation. On September 8, 2021, after internal and peer reviews, IEG produced a draft final report and shared it, in line with IEG–Bank Group protocol, with Bank Group management for their factual review and comment. On September 16, 2021, Bank Group management announced its decision to discontinue the DB report and released an external audit carried out by the law firm WilmerHale. viii On September 20, 2021, the World Bank released the Development Econom- ics External Panel Review final report. From Recommendations to Lessons The September 16, 2021, Bank Group’s management statement said, “Going forward, we will be working on a new approach to assessing the business and investment climate.” In this context and given the use of multiple other global indicators in reforms, the learning from this evaluation report is high- ly relevant. The following generalized lessons can be drawn from IEG’s evaluation report: Lesson 1: Recognizing the powerful motivational effect of reform indicators, especially those that facilitate country rankings, this evaluation notes the limitations in the coverage and guidance offered by any single indicator set on its own and advocates integrating them with complementary analytic tools and indicators. Lesson 2: Recognizing the granularity and specificity of individual reforms in any given country context, the findings from this evaluation suggest that it is better to avoid using business regulatory or similar global indicators as explicit reform objectives or monitoring indicators in Bank Group projects and country strategies focused on improving the business environment. This does not preclude the use of primary data to agreed targets that track and measure critical Bank Group institutional commitments. Independent Evaluation Group World Bank Group    ix Lesson 3: Global indicators coverage and specifications are improved if, at regular and predictable intervals, they are updated to reflect learning from research and field experience to (i) improve links to important develop- ment outcomes; (ii) strengthen relevance to the experience of the subject of coverage; and (iii) adapt to technological changes in the areas covered by the indicators. Lesson 4: The DB experience indicates the need for mechanisms and safe- guards to assure the accuracy and validity of Bank Group global indicator– based reports and related communications, using robust and transparent standards of evidence. The ultimate outcome sought with this set of lessons is to build on the many good practices observed during this evaluation; to assure that any new approach using evidence-based global indicators considers their substan- tial power to motivate and engage client countries in business environment reform; and to ensure they are used in a balanced and accurate manner that guides the choice of reform priorities with the greatest development benefits for their socioeconomic situation. The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Cover Note x Overview The Committee on Development Effectiveness requested that the Indepen- dent Evaluation Group (IEG) assess the effectiveness of using Doing Business (DB) and its strategic relevance to countries’ reform priorities and the World Bank Group’s development agenda. It parallels work by the Bank Group’s Group Internal Audit Vice Presidency on process and data integrity (World Bank 2020) and an external expert review commissioned by the Development Economics Vice Presidency focused on methodology. Since 2003, the Bank Group’s annual DB report and indicators have aimed at providing objective annual measures of business regulations and their enforcement across most of the world’s economies. Motivation, Scope, and Objectives DB indicators are considered a guide to the legal and regulatory framework for a country’s small and medium enterprises (SMEs). Their design covers the assumed key elements of a prototypical SME business life cycle, from start-up through operation to exit. Reporting annually, DB has expanded over time from five initial indicators covering 133 countries in DB2004 to 12 indicators covering 190 countries in DB2020 (figure O.1). Independent Evaluation Group World Bank Group    xi An overall ease of doing business score and ranking aggregates the scores for 10 areas, which it claims assesses “the absolute level of regulatory perfor- mance and how it improves over time” (World Bank 2019a, 19) by “benchmark- ing 190 economies to the regulatory best practice” (iii). DB’s stated aims are to (i) “motivate reforms through country benchmarking”; (ii) “inform the design of reforms”; (iii) “enrich international initiatives on development effectiveness”; and (iv) “inform theory” (World Bank 2003a, ix). Figure O.1. Doing Business Indicator Expansion over Time DB2004 DB2005 DB2006 DB2010 DB2020 1. Starting a 6. Registering 7. Protecting 11. Getting 12. (Contracting business property minority electricity with the investors government) 2. (Employing workers) 8. Paying taxes 3. Enforcing 9. Trading across contracts borders 4. Getting credit 10. Dealing with constructions 5. Resolving permits insolvency Source: Independent Evaluation Group. The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Overview Note: Indicators in parentheses are not included in the composite ease of doing business index. DB = Doing Business. Influence and Controversy DB is highly influential: it is one of the Bank Group’s most widely read publi- cations and most used set of indicators on business regulation. Country governments use DB in their developmental strategies and pro- grams. The Bank Group uses DB in (i) country strategy and policy dialogue; (ii) operations (both financing and advisory); and (iii) research and global knowledge sharing. Researchers use DB to assess the relationship of legal and regulatory conditions to various reform outcomes, to develop new indi- cators, and to test and critique the indicators. DB is also one of the Bank Group’s highly controversial undertakings. It has been the subject of multiple reviews since its inception. Controversies have arisen surrounding its methodology, accuracy, and potential biases and the way DB indicators are used in shaping and assessing country policy reforms. Methodology This evaluation assesses the relevance of DB (“doing the right things”) and its effectiveness (“doing things right”) in motivating countries to reform their legal and regulatory environment for business and identifying areas xii for reform. It uses a combination of standard evaluation methods and newer ones to gather and triangulate evidence at multiple levels: global research, country experience, and project performance. Its scope, methods, and lines of inquiry were established in a prior Approach Paper (World Bank 2021b) and Issues Paper (World Bank 2022). The Relevance of Doing Business: Is It “Doing the Right Things”? DB is part of Bank Group efforts to produce reliable, relevant, and compara- ble data and analysis on private sector development; to promote job creation, economic productivity, and gender equality; and to encourage and guide social and economic reforms promoting an efficient and fair business environment. Every year, since 2003, the DB team within the Global Indicators Group of the Development Economics Vice Presidency issues its flagship DB report. DB has actively promoted competition among countries, celebrating up through the 2020 edition “top reformers”: countries achieving the largest number of measured reforms. Country Reforms DB’s ranking encourages competition among countries and motivates gov- ernments to consider reforms. It is often a first point of engagement between Independent Evaluation Group World Bank Group    xiii country leaders and the Bank Group on addressing legal and regulatory constraints to businesses. Bank Group client countries have made substan- tial efforts to design and implement reforms measured by DB, often creating or empowering coordinating agencies to lead the reform agenda. Among the most common have been reforms to starting a business; paying taxes; getting credit; registering property; and trading across borders. DB-informed targets and reforms can feed into national development strategies and lead- ership initiatives. In many countries, governments create or empower coordinating agencies explicitly focused on the DB reform agenda. In multiple countries, the agen- cies and capacities created or empowered to pursue DB reforms later turn to a broader or deeper agenda of business environment reform, creating a “spillover effect.” Although countries are often motivated by DB, other analytic tools, indica- tors, and expertise are also mobilized to guide and deepen reforms. In many cases, reform momentum and capacity instigated by DB are applied to addi- tional reform subjects. For example, “national”-level reforms, although often initially pursued in leading cities captured by DB, are frequently extended to subnational levels. However, several country case studies reveal serious limitations to the DB in- dicators and agenda in capturing business environment reform priorities. In particular, the relevance of the DB agenda is weak in countries where struc- tural or institutional factors act as binding constraints. In other countries, it is uncertain whether the domestic SME focus of the DB agenda is the real The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Overview focus of the reform efforts. Over time, the DB agenda can lose its relevance when (i) non-DB constraints become binding after early DB reforms; (ii) pending DB reforms prove less tractable; or (iii) a DB indicator does not adapt to changes in the underlying business process or technology. Business Areas The overall ease of doing business score serves as a general index of the reg- ulatory environment. The ordering of reform priorities using DB indicators shows a relatively strong match to that of enterprise surveys within common areas of coverage. However, DB indicators do not effectively capture the real conditions expe- rienced by businesses within the business area they cover. This is generally attributed to shortcomings in the defined topical coverage of the indicators and in the representativeness of the base case scenario. Lack of substantive coverage can limit the extent to which the DB indicators can (or should) guide country reforms or reflect reform progress. The char- acteristics describing the DB “base case scenario” are not always consistent with those experienced by the typical domestic SME. xiv World Bank Group Activities DB indicators are used in Bank Group country strategies, assessments, and projects; in Bank Group client countries’ plans and reforms; and in academic research. DB is the most popular source of business environment informa- tion but not the only one, even among Bank Group products. DB is used in country strategies to identify reform needs and to motivate fu- ture operations in priority areas. Many World Bank country strategies make substantial references to DB or propose a DB-related work program or both. DB also informs a substantial share of the Bank Group’s projects that provide financing, advice, or technical assistance to client countries on the business environment. This DB-informed portfolio consisted of 676 projects repre- senting $15.5 billion in commitments during fiscal years 2010–20.1 Informed by DB, the World Bank provided lending support to 97 countries and World Bank or International Finance Corporation advisory services and analytics to 126 countries during the same period. Within the DB-informed portfolio, the most popular reform interven- tions addressed trading across borders (28 percent), ease of doing business (27 percent), and starting a business (25 percent). Project objectives focused most on improving a law or regulation (27 percent), reengineering a process (16 percent), building capacity and training (15 percent), and conducting a diagnostic (13 percent). Independent Evaluation Group World Bank Group    xv DB rankings are clearly motivational, facilitating Bank Group engagement with client countries on the business environment. However, DB indicators also have some notable inconsistencies with other Bank Group and global indicators. Many experts find the DB indicators more useful to initiate a country engagement than as explicit objectives and monitoring tools due to limited relevance and granularity. Research and Evolution Academics have found great utility in the DB indicators, producing thou- sands of articles examining the reform areas covered by the indicators and often using them to test hypotheses or construct new indexes. In several areas, there are no comparable annually updated data available for so many countries. The evaluation team found an extensive body of research literature that uses or focuses on DB indicators. A review of DB’s own database from 100 top aca- demic journals indicates that research has concentrated in a few areas, with disproportionate attention to starting a business, trading across borders, and protecting minority investors. IEG’s structured literature review points to similar clustering and gaps in methodologically rigorous research. DB indicators have evolved, including the introduction of new indicators and revisions to the methodology of existing ones. Changes of indicators are a necessary way to reflect learning and evidence about their relevance. Yet The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Overview even with such positive changes, additional modifications should be incor- porated into future revisions of the indicators to enhance their relevance to country reforms. Each change bears a cost, so it matters how they are intro- duced. However, if such changes are infrequent and well communicated, they can serve to enhance the value of the DB indicator set. The Effectiveness of Doing Business: Is It “Doing Things Right”? Activities informed by DB vary markedly in terms of the degree to which they yield intended objectives and results—their immediate and intermediate outcomes. The evaluation examined evidence of effectiveness from several sources: research (academic and professional literature), DB report claims, country reform experience, and Bank Group activity (country strategies and projects informed by DB). Research Evidence IEG’s desk review of literature collected from leading journals shows limited replicated evidence of the relationship of areas covered by DB to outcomes; more evidence confirmed by a single finding; some areas of contradictory evidence in the literature; and some gaps. IEG’s structured literature review applied rigorous criteria limiting methodology and scope to confirm sever- xvi al relationships between laws or regulations tracked by DB and significant development outcomes, while raising some cautions and pointing to gaps. For example, the structured literature review found no rigorous empirical support for outcomes linked to the DB measures of dealing with construction permits, or getting electricity. It also raises some cautions about unintended consequences of reforms measured by DB. Doing Business Report Claims Doing Business reports make numerous claims linking reforms tracked by its indicators to outcomes, led by job creation and economic growth. The claims fre- quently associate the DB reform agenda with the Bank Group’s institutional twin goals of ending extreme poverty and promoting shared prosperity. Although some of the claims contained in DB reports meet a high standard of evidence, most claims do not meet such standards, which can create a reputa- tional risk to the Bank Group. Only a minority of those claims can be confirmed through DB’s literature database, and only a very small minority of those claims have been either replicated or confirmed using a rigorous methodology. Country Reforms Experience Although many countries, donors, and even Bank Group projects use move- ment in indicators as an outcome measure, such movements are inconsis- Independent Evaluation Group World Bank Group    xvii tently linked to both reform implementation and economic outcomes. Some countries show little increase in investment, employment, or productivity after a positive movement in indicators. Overall, IEG finds a mixed picture of the links between indicators and out- comes. Deep dives carried out for five DB indicators (starting a business, getting credit, trading across borders, dealing with construction permits, paying taxes) found mixed effectiveness in specific indicator areas. IEG’s own econometric analysis suggests that it is difficult to find significant, systematic relationships between changes in DB indicators and measurable outcomes, such as gross domestic product growth, employment, foreign direct investment, trade, or labor productivity. World Bank Group Activity Experience Evidence from IEG-evaluated country strategies indicates that, of those that proposed a DB-related work program, 74 percent achieved or mostly achieved the DB corresponding objectives. However, only 45 percent also showed improvements in DB indicators. In some countries, limited impact is tied to failure to address binding constraints. IEG’s 10 case studies show both strong movement of DB indicators and tenuous links to measurable development outcomes. The DB-informed project portfolio is generally successful in achieving stated project objectives. Success rates are lower in 3 of 12 specific indicator areas requiring deep institutional reforms: enforcing contracts, registering prop- The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Overview erty, and resolving insolvency. Although client commitment and capacity are important external factors of project success, most success factors are inter- nal and largely within the Bank Group’s own control; these include analytic work, client engagement and follow-up, effective coordination, and monitor- ing and evaluation. Improving the Use of Doing Business in Country Reforms DB indicators are most relevant to motivating countries to reform their legal and regulatory environment for business and pointing to areas for reform within each indicator’s coverage area. They are best used in conjunction with complementary analysis and indicators that assure limited development resources are focused on binding constraints. DB indicators are less relevant as project-level objectives or success metrics. Given limited country reform bandwidth and resources, it is vital to contex- tualize the strongly motivating messages of DB rankings with complementa- ry sources of information to guide country reform priorities. Recommendation 1. In line with much existing practice, the Bank Group should continue to use DB to motivate client engagement and to assist in reform focus within its menu of regulatory areas–but only where the priority and nature of reforms are confirmed by complementary analytics. xviii Although the available evidence on the benefits of improving conditions measured by DB is mostly positive for development outcomes, strong evi- dence is limited by indicator area and type of reform. Recommendation 2. Consistent with good practice, the Bank Group should avoid using DB indicators as explicit reform objectives or monitoring indica- tors in projects and country strategies and, where their use is unavoidable, should not use DB as primary indicators of reform progress. DB indicators currently suffer from inadequate feedback loops between research and field experience, and between their design and application. Given its influ- ence, it is desirable for the DB approach to capture a fuller range of regulatory, legal, and institutional conditions that influence the life cycle of enterprises. Recommendation 3. The Bank Group should update DB indicator areas and definitions at regular and predictable intervals to reflect learning from research and field experience to improve links to important development outcomes, strengthen relevance to the experience of domestic SMEs and adapt to technological changes in the areas covered by the indicators. DB reports have made many claims for the benefits of measured reforms that go beyond rigorous or replicated evidence. Recommendation 4. The Bank Group should strengthen the accuracy and validity of DB claims in DB reports and related communications in line with Independent Evaluation Group World Bank Group    xix robust evidence. The ultimate outcome sought with this set of recommendations is to build on the many good practices observed in the course of this evaluation. In doing so, the Bank Group could ensure the DB indicators maintain their substantial power of motivating and engaging client countries in business environment reform in a manner that guides clients to prioritize the reforms with the greatest development benefits for their socioeconomic situation, based on a balanced and accurate consideration of evidence. 1  The portfolio includes projects that use Doing Business in their Board documents to justify the project, have one or more DB indicators in either their objectives or monitoring indicators, or are intended specifically to inform DB indicators. The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Overview xx Management Response Management of the World Bank thanks the Independent Evaluation Group for sharing the report, The Development Effectiveness of the Use of Doing Busi- ness Indicators, Fiscal Years 2010–20, along with the cover note. Management notes that this evaluation was substantially completed before management took its decision—on September 16, 2021—to discontinue the Doing Business report. The World Bank Group will actively continue its work to advance the role of the private sector in development and will work through multiple channels to help governments design a business environment that supports this objective. Since mid-2020, the Bank Group has undertaken several re- views and audits of the Doing Business report and its methodology, process, and governance. Independent Evaluation Group’s findings, and the findings of other reviews and audits, will be taken into consideration as management develops new approaches to assessing the business and investment climate to inform its support for private sector development. International Finance Corporation Management Response The management of the International Finance Corporation (IFC) welcomes Independent Evaluation Group World Bank Group    xxi the Independent Evaluation Group’s evaluation of the Doing Business report and its summary of the lessons drawn from this experience for the World Bank Group. As users of the indicators, IFC has also benefited from the eval- uation and has identified a few lessons for its future application and use of global data sets: » Global indicator data sets can be very good door-openers for the discussion on key policy issues. These cross-country comparisons offer very useful information and incentives for governments. IFC management welcomes the investment in building such data sets, despite the limitations they may carry. » For IFC, it is important to ensure that the teams using these data sets are fully aware of the limitations in scope, as they apply, and ensure that planned activities are set within the broader context of the World Bank Group country strategies and the upstream and investment climate focus of the activities. » Teams will also be selective in using the relevant indicators from these data- sets as monitoring and evaluation indicators. The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Report to the Board xxii Report to the Board from the Committee on Development Effectiveness The Committee on Development Effectiveness met to consider the report by the Independent Evaluation Group (IEG) The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20, the World Bank management response, and the International Finance Corporation manage- ment response. The committee commended IEG for the comprehensive evaluation and ap- preciated IEG’s commitment to review and correct any facts or misrepresen- tations in the draft before its public disclosure. IEG noted the implications of its findings and recommendations for new business environment indicators, including strongly connecting the design of new indicators and diagnostics to research and country experience, using new indicators alongside com- plementary analysis to help inform well-designed and comprehensive re- forms focused on binding constraints rather than as ends of themselves, and ensuring that claims linking reforms captured by the indicators are validated by robust evidence. Members noted that although the Doing Business (DB) Independent Evaluation Group World Bank Group    xxiii flagship report has been discontinued, enabling a business environment re- mains a relevant task for the World Bank and the International Finance Cor- poration. Important lessons could be drawn from the rich and high-quality analysis to shape the strategic focus and design of the new approach, includ- ing connecting policy advice more strongly to research and data and de- ploying appropriate safeguards to assure accuracy and validity of indicators, focusing on statutory regulations along with their actual implementation. Therefore, they encouraged management to deploy appropriate mechanisms and safeguards to assure the accuracy and validity of global indicator-based reports using robust and transparent standards of evidence and that indica- tors should be considered and interpreted along with complementary diag- nostics and analytics, updated periodically and tailored to country contexts. Members stressed that the evaluation’s findings and lessons on the creation and use of indicators were still timely and should thus be used to shape the strategic focus and design of the World Bank’s new approach to global indi- cators on business reform and investment climate. Given that the DB was one of the Bank Group’s most widely read publica- tions and most used set of indicators on business regulations to motivate clients to engage in business environment reforms, members and nonmem- bers reiterated that the process of designing a successor to the DB should be comprehensive and stressed the need for management to consult the Board of Executive Directors and client countries. They highlighted that the original objectives of the DB remain relevant and encouraged management to incor- porate in the new approach the positive features found in the evaluation as well as IEG’s conclusions and the conclusions of other recent evaluations on the DB, in particular on validity, relevance, and the need to focus more on binding constraints, sustainability, and issues such as working conditions. Members acknowledged that the engagement with the Board on the new ap- proach would begin with the informal meeting scheduled on January 18, 2022. Members agreed that movements in indicators should be interpreted careful- ly and encouraged management to ensure that country strategies, policies, and development agendas reported reform implementation accurately. They suggested that complementary sources of data, as well as analytical guid- ance, should be used to validate the relevance of the reform topics. Acknowl- edging that the DB indicators had had strong weight on legal information (according to statutory laws and regulations), management reassured mem- bers that the new approach would seek for better balance with actual indi- cators, including surveys in the field, for a better match between regulations and business practice. 1 | Introduction: Doing Business and Country Reforms Highlights Since 2003, the annual Doing Business (DB) report and indicators have aimed at providing objective measures of business regu- lations and their enforcement as they affect small and medium enterprises across most of the world’s economies. DB is highly influential. It is one of the World Bank Group’s most widely read publications and offers the most used set of indicators on business regulation. Country governments use DB in their developmental strategies and programs. The Bank Group uses DB widely, including in (i) country strategy and policy dialogue; (ii) operations (financing and advisory); and (iii) research and knowledge sharing. Researchers use DB to examine the relationship of conditions it measures to reform outcomes. DB is highly controversial, having been the subject of multiple past reviews that revealed limitations in the indicators’ methodology, accuracy, potential biases, and the way they are used in shaping and assessing country policy reforms.   1 Background and Context The Committee on Development Effectiveness of the World Bank Group Board of Executive Directors requested that the Independent Evaluation Group (IEG) carry out an independent evaluation of Doing Business (DB) to assess DB’s strategic relevance and effectiveness to client countries’ reform priorities and to the Bank Group’s strategic agenda. This evaluation con- siders how DB is used to guide business environment reforms—both those supported by the Bank Group and those undertaken without its support. It focuses on the influence of DB on country reforms through IEG’s mandat- ed lens of development effectiveness. It parallels work by the Bank Group’s Group Internal Audit Vice Presidency on process and data integrity and an The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Chapter 1 external expert review commissioned by the Development Economics Vice Presidency (DEC) unit focused on methodology. IEG, DEC, and the Group Internal Audit Vice Presidency shared approaches and early findings during the life span of this work. An Issues Paper reviewed existing evidence to identify key issues about the use and influence of DB indicators and their relevance to country policy reforms (World Bank 2022). Since 2003, the annual DB report and indicators have aimed at providing objective annual measures of a set of business regulations and their enforce- ment across most of the world’s economies.1 From the start, DB was designed with an ambitious set of stated aims, intending to capture “the efficiency and strength of laws, regulations and institutions relevant to domestic small and medium-size companies throughout their life cycle.”2 DB indicators are presented as a guide to the legal and regulatory framework for small and medium enterprises (SMEs), intending to cover the key ele- ments of a prototypical business life cycle from start-up to operation to exit. DB has expanded over time from 5 initial indicators covering 133 countries in DB2004 to 12 indicators (10 of which feed an overall ease of doing busi- ness [EoDB] index) reported annually and covering 190 countries in DB2020 (box 1.1). From the start, DB intended to: i. Mobilize demand for reform of the business regulatory environment; 2 ii. Inform the design of reforms (DB wanted to target “what needs to be changed” and disseminate “the experience of countries that perform well according to the indicators”; World Bank 2003a, ix); iii. Enrich international initiatives on development effectiveness. “[R]ecog- nizing that aid works best in good institutional environments,” DB aimed at explicitly allowing donors to base aid on “good-quality data that can be influenced directly by policy reform” (World Bank 2003a, x); and iv. Inform theory (DB intended to inform the field of “regulatory economics” by facilitating “tests of existing theories” and contributing to “the empir- ical foundation for new theoretical work on the relation between regula- tion and development; World Bank 2003a, x). Influence DB is recognized as highly influential in business regulatory reform world- wide, and it is the most used set of indicators on business regulation. Its reports are among the Bank Group’s most widely read publications,3 its web- site among those the most visited. Among global indicators of the business environment, it has been estimated to hold a 65 percent market share.4 As elaborated in chapter 2, many country governments use DB in their develop- mental strategies and programs. The Bank Group uses DB widely, including in (i) country strategy and policy dialogue, (ii) operations (both financing and advisory), and (iii) research Independent Evaluation Group World Bank Group    3 and global knowledge sharing. Its indicators are also widely used and analyzed in the academic literature. In addition, they are a component of many other influential indexes, including the World Economic Forum’s Global Competitiveness Index, the Heritage Foundation Index of Economic Freedom, and the Fraser Institute Economic Freedom Index. Further, DB has also been influential as an approach to indicators (box 1.1), generating spin-offs like subnational DB; Women, Business, and the Law; and Enabling the Business of Agriculture. Box 1.1. What Are the Doing Business Indicators? Doing Business (DB) is presented as a guide to the legal and regulatory framework for small and medium enterprises. The annual DB has expanded over time from initially reporting on 5 indicators covering 133 countries in DB2004 to reporting on 12 indicators (10 of which feed an overall ease of doing business [EoDB] index) covering 190 coun- tries in DB2020 (figure B1.1.1). Figure B1.1.1. Evolution of Doing Business Indicators DB2004 DB2005 DB2006 DB2010 DB2020 The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Chapter 1 1. Starting a 6. Registering 7. Protecting 11. Getting 12. (Contracting business property minority electricity with the investors government) 2. (Employing workers) 8. Paying taxes 3. Enforcing 9. Trading across contracts borders 4. Getting credit 10. Dealing with constructions 5. Resolving permits insolvency Source: Independent Evaluation Group. Note: Indicators in parentheses are not included in the composite ease of doing business index. DB = Doing Business. The indicators are intended to cover the key elements of a prototypical business life cycle from start-up to operation to exit. DB also provides an overall EoDB score and ranking by aggregating the scores for 10 areas, which it claims assesses “the absolute level of regulatory performance over time” (World Bank 2019a, 19) by “benchmark[ing] economies with respect to regulatory best practice” (World Bank 2019a, 77). The methodology collects data from informed experts who are generally intermediaries for businesses in dealing with a given area of regulation or law, such as lawyers, accoun- tants, or freight forwarders. It uses standardized case “scenarios” to enhance country comparability. Case scenarios generally apply to a country’s largest or two largest business cities (in countries with over 100 million people, since 2015). Some cases are otherwise limited. For example, dealing with construction permits looks only at proce- dures associated with a warehouse of specified characteristics. The use of case sce- narios and intermediary informants has contributed to the ability of DB, until DB2020, 4 (continued) Box 1.1. What Are the Doing Business Indicators? to publish 17 annual volumes covering an expanding majority of countries within a budget acceptable to the Bank Group. The indicators themselves have evolved over time, with some experiencing multiple revisions to methodology or components and the introduction of new indicators. The DB indicators are composed of a mix of three different types of subindicators. The first type, “less is better” indicators, measure time, cost, procedures, or docu- ments required for regulatory compliance. The second type measures the “presence of a feature,” which can be legal, regulatory, or institutional. The third type measures “outputs or outcomes” of processes that are observed in practice. Among the EoDB indexed indicators, “less is more” indicators constitute 62.5 percent of total weight in the EoDB score, “presence of a feature” indicators constitute 25.6 percent, and “output or outcome” measures constitute 11.9 percent of the score. Source: World Bank 2009, 2010, 2011, 2012, 2013, 2014a, 2015a, 2016c, 2017b, 2018b, 2019a. The use of the indicators is intended to link to positive development out- comes through the information and influence of the indicators on multiple key actors, leading to beneficial country reforms and development outcomes. The implicit theory of change for this evaluation reflects this hypothesized influence model (figure 1.1). Channels of influence range from direct influ- ence on government leaders to indirect influence through donors, investors, Independent Evaluation Group World Bank Group    5 civil society, and the press. Country governments also use DB in their development strategies and programs. Many governments establish explicit targets for progressing in their DB rank- ing, and many more incorporate reforms associated with improvements in DB scores into their development plans. Some of these targets are included in the World Bank’s Country Partnership Frameworks. From DB2010 through DB2020, DB tracked 2,960 positive, country-level business regulatory reforms across 184 economies (of 190 measured). DB has explicitly encouraged competition among countries on measured reforms, celebrating top reformers. The Bank Group uses DB widely. DB indicators are used in country strategies and policy dialogue to describe business-enabling conditions in a country, or to motivate future operations in priority areas identified in Country Part- nership Frameworks, for example, or in diagnosing private sector development limitations in Country Private Sector Diagnostics (CPSDs). DB is also used some- times to justify Bank Group lending operations or advisory services, to influence their design, to target certain projects, or to assess the progress of some projects against agreed reform objectives. For example, IEG estimates that, during the fis- cal year (FY)10–20 evaluation period, about 64 percent of country strategies and 676 Bank Group projects were informed by DB indicators. Finally, the Bank Group also uses DB indicators for research and to share global knowledge.  ow Doing Business Influences Actions Leading to Figure 1.1. H Development Outcomes Intermediate The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Chapter 1 Donors Outcomes (including Enterprise & World Bank Level rm ce Group) fo n Reduce: In flue In Compliance Influence cost, time, Advise Immediate procedures, Finance Outputs Outcomes uncertainty Impacts Inform & Increase Influence Government Legal, Legal, Jobs, Officials/ Regulatory regulatory, Investment, Leaders Design and and service Growth, Implement Procedural quality Poverty Influence Reforms Reforms Alleviation, Finance Shared Inf Economy Prosperity o International Infl rm & Level ue and Domestic nc Reduce: e Investors Deadweight loss, Civil Society, discretion Press Increase Regulatory compliance, efficiency, market dynamism Source: Independent Evaluation Group; Doing Business website (https://www.doingbusiness.org/en/ doingbusiness); review of World Bank Group literature; Kelley and Simmons 2020. Researchers use DB to study the outcomes of regulatory design, burden, and reform. DB reports that think tanks and research organizations use the indica- tors “both for the development of new indexes and to produce research papers” (World Bank 2016, 21) regarding the relationship of business regulation to economic outcomes. In several areas, there are no comparable annually updat- ed data for so many countries. Although DB’s relevance to research is hard to quantify, the DB team itself collected over 400 articles from 100 leading journals of relevance to DB, and IEG’s own structured literature review (SLR) found close 6 to 1,900 articles of potential relevance based on search terms and a review of abstracts relevant to 10 DB indicator areas. DB2019 reported that there have been “more than 3,400 research articles discussing how regulation in the areas mea- sured by Doing Business influence[s] economic outcomes” published in peer-re- viewed academic journals, 1,360 of those published in the top 100 journals, and another 9,450 “published as working papers, books, reports, dissertations or research notes”(World Bank 2018b, 32). Controversy DB stands out as one of the Bank Group’s controversial undertakings. Con- troversies arose concerning the indicators’ methodology, accuracy, and potential biases and the way they are used in shaping and assessing country policy reforms. The Bank Group and IEG have reviewed DB several times in the past, largely to respond to such criticisms. Past reviews recommended continuation of the indicators but also identified flaws and recommended modification of their design, collection, or application (table 1.1). Table 1.1. Findings of Past Evaluations and Reviews of Doing Business Positive Findings Negative Findings Influence and use Limitations of scope and coverage » “The DB indicators have motivated » With significant exceptions, DB indicators policy makers to discuss and consider focus on the compliance costs of business business regulation issues. Its active regulations—time, money, or procedural Independent Evaluation Group World Bank Group    7 dissemination in easy-to-understand steps—where less regulation is better. language permits widespread press Exceptions include the extent of protec- coverage and generates interest from tion of property, the protection of minority businesses, nongovernmental orga- shareholder rights, an index of building nizations (NGOs), and senior policy quality control and the range of assets that makers” (World Bank 2008, xvi). can be used as collateral. It was proposed » DB is “transformational, as it fundamen- that the Bank Group develop “separate tally changed the way the quality of measures (…) to capture a wider range of the business environment is measured, benefits and costs (social, economic, and and it catalyzed actions to address environmental) if existing regulations are constraints” (World Bank 2016d, 8). changed” (World Bank 2015b, xl; 2019b). » DB is “influential” and “widely used” and, » On paying taxes, IEG and the Independent although controversial, attracting “un- Review Panel recommended excluding rivaled media and high-level attention” the tax rate from the indicator, stating (World Bank 2018f, 7–8). that it was more a matter of public policy than compliance burden (World Bank 2008; Doing Business Report Independent Review Panel 2013). (continued) Positive Findings Negative Findings Usefulness to guide or monitor reforms Limits in tracking reform or monitoring impact » “Within the limits of the Doing Business » Doing Business indicators track the inci- indicators, most investment climate in- dence of reforms but not the quality or terventions produce positive interme- impact of reforms. IEG recommended diate outcomes in terms of improve- that the World Bank Group trace coun- ment in time, number of procedures, try-level impacts of DB reforms (World and cost” (World Bank 2015b, 67). Bank 2008). » IEG finds a positive link between the » The Independent Review Panel criticized World Bank Group’s support for trade DB’s implied claim that the reforms it facilitation reform and improved Doing promotes cause economic development Business trade indicators (World Bank and growth, stating “correlations do not 2019b) justify a causal interpretation” (Doing The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Chapter 1 Business Report Independent Review Panel 2013, 2). Value meriting retention Questions on aggregation » The Independent Panel Review » The Independent Review Panel found recommended that “the Doing Busi- that aggregating rankings across topics ness report be retained as an annual is challenging because it explicitly or flagship report” (Doing Business Report implicitly involves a value judgment Independent Review Panel 2013, 4). of what is “better” for doing business » The ease of doing business indicators and how much better it is. It also found “are one of the World Bank’s most im- that “small revisions or inaccuracies in portant contributions to research and primary data can significantly change public policy” (Morck and Shou 2018, 3) a country’s rankings” (Doing Business Report Independent Review Panel 2013, 3). It recommended that the World Bank publish the DB report without the overall aggregate rankings (that is, without the ease of doing business ranking). Discontinuities » “Frequent methodology changes reduce the value of the indicators to research- ers, policy makers and the media” (Morck and Shou 2018, 1); “The World Bank may wish to minimize methodology changes except to fix confirmed problems with existing methodology” (Morck and Shou 2018, 2). Sources: Doing Business Report Independent Review Panel 2013; Morck and Shou 2018; World Bank 2008, 2015b, 2016d, 2019b. Note: DB = Doing Business; IEG = Independent Evaluation Group. 8 Evaluation Approach To evaluate the use of DB and its relationship to country reforms, IEG first developed an Approach Paper detailing the objective, scope, and proposed evaluation methods. Several lines of inquiry were later identified in an Issues Paper (World Bank forthcoming; appendix A). This evaluation used several standard and some relatively new evaluative methods to gather information and triangulate evidence at the global, country, and project levels (appen- dix A). Innovations included employing supervised machine learning to support the identification of projects for the portfolio review and analysis (appendix B) and the identification of impact-related claims made within the DB reports (appendix J). Using a Bank Group DB-informed portfolio and DB- tracked reforms, the evaluation team developed statistical and econometric analyses (appendix H) to explore project and country outcomes. Because of the focus on country reforms, the team undertook 10 country case stud- ies covering countries of diverse characteristics in terms of region, income level, fragility, level of engagement with the Bank Group, and number of DB-influenced reforms (appendix C). The team also developed five indicator area deep dives (appendix D). Due to the COVID-19 pandemic, almost all fieldwork for both case studies and deep dives was conducted virtually. The literature review included both DB’s own bibliography and a rigorous SLR (appendixes E and F). The team also reviewed country strategies (appen- dix G) as well as Project Performance Assessment Reports and International Independent Evaluation Group World Bank Group    9 Finance Corporation (IFC) advisory services (AS; appendix I). 1  Doing Business (DB) came out of a strong focus on the investment climate in the early 2000s. In a 2002 volume, World Bank chief economist Nicholas Stern stated: “The central challenge in reaping greater benefits from globalization lies in improving the investment climate—that is, in providing sound regulation of industry, including the promotion of competition; in overcoming bureaucratic delay and inefficiency; in fighting corruption; and in improving the quality of infrastructure. While the investment climate is clearly important for large, formal sector firms, it is just as important—if not more so—for small and medium-size enterprises, the informal sector, agricultural productivity, and the generation of off-farm employment. For these reasons, the investment climate is itself a key issue for poverty reduction” (Stern 2002, 52). This focus led to a strong drive for a systematic approach to diagnostics and results mea- surement in the area of investment climate to help “clients understand investment climate conditions in their country in comparative light and induce … action to build a sound policy The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Chapter 1 and institutional environment for investment” (World Bank 2003b, 2). Even at its year of in- ception, DB was acknowledged to be “a cost-effective way to assess certain dimensions of the investment climate” focusing on “constraints faced by businesses (such as basic incorporation procedures)” that could be “captured effectively and inexpensively by interviewing relevant experts” (World Bank 2003b, 4). 2  “Global Indicators Group” page of Doing Business website, under heading “Doing Business,” https://www.doingbusiness.org/en/about-us/global-indicators. 3  Each report not only updates the national indicators but also highlights an issue of interest, in some way related to DB. Each volume reports on trends, highlights top reformers and mod- el reforms, and cites evidence for the importance of DB overall and for the importance of the individual indicators and reform areas covered. With titles like Smarter Regulations for Small and Medium Enterprises (World Bank 2012) and Reforming to Create Jobs (World Bank 2017b), they make a case for the DB agenda. 4  Doshi, Kelley, and Simmons 2019; Besley 2015; Haidar 2012, attributed to Harvard Berkman Center, “Media Cloud Database,” 2017. 10 2 | The Relevance of Doing Business: Is It “Doing the Right Things”? Highlights Doing Business’s (DB’s) ranking encourages competition among countries and motivates governments to consider reforms. DB has often been a first point of engagement of client countries in ad- dressing legal and regulatory constraints to businesses. Motivated by DB, many client countries have launched reform initia- tives, with coordinating agencies leading the reform agenda. In sev- eral countries, those agencies have later pursued broader or deeper reforms not captured by the DB indicators (spillover effects). Although DB rankings facilitate World Bank Group engagement related to the business environment, lack of granularity limits the indicators’ ability to monitor and evaluate reforms. DB indicators are perceived to be more useful for initial policy dialogue and en- gagement than as explicit objectives and monitoring tools. DB indicators have serious limitations in capturing binding business environment constraints. The ability of specific individual indicators to reflect the specific business areas they cover is also limited. The relevance of DB indicators can be constrained by unaddressed structural and institutional reform priorities, methodologies, and assumptions divorced from the reality faced by local small and medium enterprises. DB indicators have notable inconsistencies with other Bank Group and global indicators.   11 DB is part of Bank Group efforts to produce reliable, relevant, and com- parable data and analysis on private sector development. The ultimate objective is to promote job creation, economic productivity, and gender equality and to encourage and guide social and economic reforms promoting an efficient and fair business environment. Each year, the DB team within the Global Indicators Group of the DEC of the World Bank issues its annu- al flagship DB report. Claims to relevance (box 2.1)1 depend in part on the proven link of DB-inspired reforms to positive outcomes for private sector development, employment, productivity, and other beneficial outcomes ex- amined in the next chapter on effectiveness. The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Chapter 2 Box 2.1. Views of World Bank Group Expert Practitioners In IEG interviews with World Bank Group practitioners and managers, views expressed are overwhelmingly favorable with regard to the use of Doing Business (DB) to initiate dialogue and most views are positive as to DB’s relevance to country priorities (figure B2.1.1). 60 percent of views expressed in interviews identify indicators as validly mea- suring their business area, while 40 percent do not. Fewer than half of expressed views are favorable regarding DB as a guide for identifying binding constraints, and only 26 percent suggest DB indicators are a good way to monitor or evaluate reforms.  xperts’ Views of Doing Business Expressed in Semi- Figure B2.1.1. E structured Interviews 60 Favorable versus� unfavorable views (no.) 50 3 40 20 18 23 30 47 28 20 30 26 10 20 10 0 Construct validity Relevance Value to initiate Value to guide Value to monitor (degree to which they to country reform dialogue reforms by and evaluate measure what they priorities identifying those reforms claim to be measuring) that address binding constraints Evaluative content of interviews Unfavorable Favorable view 12 Source: Independent Evaluation Group. Note: Interviews were performed with 20 World Bank Group expert practitioners. Since 2003, DB has become a key resource for Bank Group work on the in- vestment climate, for client country reforms, and for research. IEG’s review suggests the DB indicators have tracked thousands of reforms in client countries. In FY10–20, they were used to inform approximately 64 percent of Bank Group country strategies and were used in an estimated 676 projects with interventions worth $15.5 billion in commitments. They also informed a large number of research articles. Client Country Reforms and Development Goals Client countries have made dedicated, substantial efforts to design and implement reforms measured by DB. Since 2010, DB has tracked 2,960 business regulatory positive, national reforms across 184 economies. Each year between 110 and 140 countries are reported to have introduced positive reforms. Among the most common have been reforms to starting a business, paying taxes, getting credit, registering property, and trading across borders (figure 2.1). Based on a 20 percent stratified random sample, the most popu- lar types of reforms recorded have been improvements to business laws and regulations (23 percent), reengineering of processes (22 percent), automa- tion or introduction of electronic systems (20 percent), reductions of fees or rates (16 percent), and establishment or reform of agencies (6 percent). Although reforms across income levels were similar, upper-middle- and high-income countries were more likely to introduce electronic or automat- ed systems, whereas low-income countries were more likely to improve laws Independent Evaluation Group World Bank Group    13 or regulations or reengineer business regulation processes. For example, DB2018 documented that Kuwait introduced online business registration, while Mozambique reduced the time to get an electricity connection by con- solidating procedures (World Bank 2017b). Fragility, conflict, and violence countries are also more likely to focus on improving laws or regulations, but they disproportionately focus on reducing fees and rates. For exam- ple, DB2019 reported that Papua New Guinea improved legal protection of minority investors, while Burundi, Myanmar, and Afghanistan reduced fees associated with starting a business (World Bank 2018b).  ositive National Reforms Tracked by DB2010–DB2020 Figure 2.1. P Employing workers (n = 91) Trading across borders 3% (n = 294) Starting a business 18% 10% (n = 516) Resolving insolvency (n = 177) 6% Protecting investors 6% (n = 185) Paying taxes 14% (n = 403) 6% Getting electricity (n = 189) 7% The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Chapter 2 12% Enforcing contracts Getting credit (n = 216) (n = 353) 8% 10% Dealing with construction permits Registering property (n = 249) (n = 287) Source: Doing Business reform database 2020. Note: Includes only positive reforms, excludes subnational reforms. DB has actively promoted competition among countries, celebrating “top reformers”—the countries achieving the largest number of measured re- forms—up through the 2020 edition. Many countries have responded with initiatives to improve their ranking: Indonesia strives to be in the top 40 countries for “ease of doing business” (EoDB), Morocco to be in the top 50, and the Russian Federation to be in the top 20; Rwanda continually drives to be recognized among the top reformers. During the evaluation period (FY10–20; in declining frequency), Rwanda, Kazakhstan, Indonesia, the Unit- ed Arab Emirates, India, Ukraine, Azerbaijan, Russia, Uzbekistan, Armenia, Brunei Darussalam, Kenya, and Morocco were credited with the highest total number of positive national reforms. DB-informed targets and reforms feed into national development strategies and leadership initiatives. Rwanda’s first Economic Development and Pover- ty Reduction Strategy (2008–12; Rwanda Ministry of Finance and Economic Planning 2007) cites a specific DB indicator, and its second strategy (2013– 18) cites a key accomplishment of the first plan: “the continued reforms 14 in the Doing Business environment has laid the foundations for Rwanda to develop into a top investment and trade destination within Africa” (Rwanda 2013, 19). In May 2012, newly reelected Russian president Vladimir Putin approved the so-called “May decrees,” ordering government to increase Russia’s ranking in the World Bank’s EoDB index from 120th place in 2011 to 50th place in 2015 and 20th place in 2018 (Presidential Decree No. 596). In Indonesia, DB strongly influenced national development planning, with President Joko Widodo in 2017 affirming his intention for Indonesia to jump to 40th place in the EoDB ranking and instructing ministries to draft detailed plans to achieve it. India and Morocco both set explicit targets to be among the top 50 countries (by EoDB rank) in the world. DB has been a first point of engagement with business environment reform for multiple countries, with reforms of legal and regulatory aspects of their business environments. IEG’s 2016 report on transformational engagements found DB to be transformational in ways other business environment diag- nostics, including enterprise surveys, were not, in part because “it catalyzed actions to address constraints” (World Bank 2016d, 22). It found that “[e] ngagements with clients through the global Doing Business appear to have found traction with clients far more frequently than country engagements through [survey-based investment climate assessments]” (22). Interviews with Bank Group expert practitioners and managers suggest this is the area where DB indicators have the greatest value—“opening the door.” Even some countries initially critical of DB, such as India, China, or Morocco, Independent Evaluation Group World Bank Group    15 have over time embraced its approach and agenda. Typically, a low ranking in the overall EoDB index or in individual indicators can capture the atten- tion of government leadership and stimulate a request to the Bank Group for an engagement, often beginning with a DB reform memorandum (box 2.2) or matrix.2 The memo lays out an agenda of changes to laws, regulations, pro- cedures, and institutions that would improve a country’s DB score in specific indicator areas. Of 10 case study countries covered in this evaluation, 8 had DB reform memos prepared and others had a reform matrix. Box 2.2. Doing Business Reform Memorandums The Independent Evaluation Group reviewed 10 Doing Business (DB) reform memoran- dums (appendix C, addendum) delivered to governments for countries featured in the Independent Evaluation Group’s case or desk studies. Although the memos contained many useful, practical recommendations and tools to help improve DB indicator scores, the recommendations rarely drew from other sources or frameworks. They therefore risked missing overall challenges in each country’s business environment and even within each DB area of focus. In multiple cases, the best practice examples used were not tailored to the recipient’s stage of development or capacity. Source: Independent Evaluation Group, review of 10 Doing Business reform memorandums.. The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Chapter 2 Even in countries with small portfolios of DB-informed Bank Group oper- ations, interaction with the Bank Group can lead to reforms. For example, both the United Arab Emirates and Russia, two countries with limited Bank Group DB-informed portfolios but a high level of reform activity, benefited from DB reform memos. The reform memo is often followed by technical assistance or financing support from the Bank Group, sometimes from other donors, and sometimes (especially for wealthier countries) from the govern- ment itself. The United Arab Emirates and Russia were both clients of World Bank reimbursable advisory services. Several governments also hired private consultants to support reform design and implementation efforts; high-in- come countries may use existing capacity. The dual role of the Bank Group as generator of indicators and supporter of reforms to improve indicators has raised questions of a conflict of interest. IEG did not find this to be a common perception of clients and stakeholders (box 2.3). 16  he Perception of World Bank Group Conflict of Interest Box 2.3. T between Indicator Generation and Operational Work A recurrent theme emerging from Independent Evaluation Group (IEG) interviews with World Bank Group experts and operational staff was a concern about conflict of inter- est. Clearly, the recent suspension of Doing Business and new safeguards agreed to after a Group Internal Audit Vice Presidency audit reflect concerns that the interest in generating operations or pleasing clients has the potential to conflict with the interest in generating impartial data. A general perception of conflict of interest could chal- lenge the credibility of the indicators or pose reputational risk to the Bank Group. Although concern for or perception of conflict of interest was reflected in multiple Bank Group interviews, IEG did not encounter it in interviewing country counterparts and stakeholders for its case studies, even in response to explicit questions on the topic. (It is not clear whether the virtual online interview format affects interviewees’ candor or perception of confidentiality.) Several clients were clearly motivated to seek engage- ment with the Bank Group by a perception that the Bank Group knew how to improve their DB scores. In countries with reimbursable advisory services, there was an implicit potential for clients to “pull the plug” on financing if indicators did not move as intended. Yet, although the potential for conflict of interest is real, IEG did not find it to negatively influence client and stakeholder perceptions of the reputation of the Bank Group. Source: Independent Evaluation Group. Independent Evaluation Group World Bank Group    17 In most cases, governments create or empower coordinating agencies ex- plicitly focused on the DB reform agenda. Typically, such agencies use DB indicators to help elaborate an initial agenda, to set explicit targets, to spur coordination and consultation, and to monitor and then publicize progress. In Morocco, for example, CNEA (Comité National de l’Environnement des Affaires; National Business Environment Committee) was formed in late 2010 as a specialized coordinating agency reporting to the prime minister. CNEA used DB indicators as a focus for reforms, to initiate areas of reform, as a clear basis to coordinate activities of diverse agencies, as metrics and monitoring indicators for reform, as the subject of public-private dialogue, and as a way to communicate to foreign investors and donors Morocco’s reform success. In Rwanda, a DB unit within the Rwanda Development Board (established in 2008) prepared an action plan on a yearly basis to be approved by a National Business Steering Committee. The unit designed and monitored DB-related reforms, managed development funds, orga- nized working groups, advised agencies, and communicated with the private sector. The steering committee oversaw implementation of reforms at the cabinet level, coordinating relevant ministries and institutions. In addition, a Parliament Economic Committee reviewed DB-related laws and could fast- track priority measures. By contrast, in 2017, Chinese authorities used the powerful Ministry of Finance to coordinate reforms both nationally and in Beijing and Shanghai (box 2.4). The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Chapter 2 Box 2.4. Doing Business Reform: Traction in China The Doing Business (DB) indicators found strong traction in China from 2017 onward. Beyond the interest in achieving rank improvement, a deeper appreciation grew of DB’s value as a diagnostic and benchmarking tool that would help cities in China assess their own performance and identify areas for change. The Ministry of Finance mobilized staff at the highest levels in Beijing and Shanghai, and city-wide reform plans were formulated using DB reform recommendations. The civil service developed a detailed knowledge of the indicators and their limitations. Both the International Finance Corporation and World Bank engaged in supporting reforms. The International Finance Corporation began early, through access to finance work greatly enhancing credit information. World Bank reimbursable advisory services (RAS) supported work on the insolvency framework. Another RAS project focused on Shanghai, seeking to improve construction permitting and trading across borders, while a Beijing-focused RAS project supported reform of construction permitting and property registration. The National Development and Reform Commission began exploring possible Chinese indicators and application outside the principal cities. Relevant areas were identified but use of formal indicators was not yet evident at the time of the case study. Source: Independent Evaluation Group In many countries, the agencies and capacities created or empowered to pur- sue DB reforms later turn to a broader or deeper agenda of business environ- ment reform, creating a “spillover effect.” Given DB’s strong role observed as 18 a “door opener” for reform activities, it is not surprising that the open door can lead to reforms not captured by the DB indicators. Such spillover ben- efits can occur through a deepening of reforms within legal and regulatory areas covered by DB, a broadening of reforms to other aspects of the busi- ness environment, or a geographic broadening of DB reforms subnationally to cities or localities not captured in the DB indicators. Clearly, where there are limits to DB’s approach to identifying reform priorities or guiding and measuring reforms, such spillovers may not be entirely positive. Although countries are often motivated by DB, other analytic tools, indica- tors, and expertise are often mobilized to guide and deepen reforms. Eight of IEG’s 10 case studies explicitly identified the use of such augmentation (box 2.5). Sometimes other indicators—such as enterprise surveys, the World Economic Forum’s Global Competitiveness Report indicators, or the Organi- sation for Economic Co-operation and Development regulatory indicators— were used, but often frameworks and tools specific to a reform area (such as the Tax Administration Diagnostic Assessment Tool) were applied. IEG’s review of IFC AS projects notes that teams often find DB indicators ill-suit- ed to guide and monitor projects, and in many cases, teams conducted their own diagnostic surveys or used other tools to generate more specific and tailored guidance and benchmarks. Bank Group projects on secured trans- actions (covered by “getting credit”) often draw on the 2007 United Nations Commission on International Trade Law (UNCITRAL) legislative guide on Independent Evaluation Group World Bank Group    19 secured transactions, the 2016 UNCITRAL Model Law, the United Nations Convention on Assignments of Receivables Accounts, in Internal Trade, or the Insolvency and Creditor Rights Standard developed by the World Bank and UNCITRAL. The reform momentum and capacity instigated by DB are often applied to additional topics. In Morocco, the CNEA in recent years has added new issues to its agenda, applying both the coordinative capacity and indicators-based reform approach to additional areas and to subnational governments not included in the first round of reforms. Under its reform process, Indonesia approved its Omnibus Law modifying over 70 laws linked to easing the policy burden on business, extending well beyond what was covered by the DB indi- cators alone.3 In 2019, Indonesia’s head of the Ministry of National Develop- ment Planning announced a transition to e-bureaucracy to support the ease of doing business, again extending beyond measures explicitly captured by the indicators. Not all countries show such spillover benefits—for example, in Afghanistan, the agenda remained focused on DB.  olombia: Broadening the Evidence Base for Reform, Fiscal Box 2.5. C Years 2010–20 Colombia’s regulatory reform objectives were informed by Doing Business (DB) and complemented by other analytical work. DB’s scope was too narrow to identify reform priorities by itself and was used in combination with other tools and sources of data. The government of Colombia regularly monitors three indicator reports along with The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Chapter 2 sector-specific data and research: Doing Business, the World Economic Forum’s Global Competitiveness Report, and the International Institute for Management Development’s World Competitiveness Yearbook. Colombia used DB data as supporting evidence for its business environment diagnoses and, in some cases, to point to areas in need of reform. To complement DB data, the government of Colombia closely monitors the World Economic Forum’s Global Competitiveness Report series, referencing the reports in national development plans and utilizing their insights related to the government of Colombia’s overall competitiveness strategy beyond legal and regulatory aspects. The Organisation for Economic Co-operation and Development has also been a trusted adviser, providing information on best practices and benchmark data on both general competitiveness and productivity issues and on regulatory reform. It closely advised Colombia’s reforms to the regulatory process and produced regulatory reviews in the context of Colombia’s accession to Organisation for Economic Co-operation and Development. The World Bank’s own support to reform through development policy loans and advisory services was based in broader analysis than DB, reflected in policy notes, working papers, and the Systematic Country Diagnostic. Source: Independent Evaluation Group. National-level reforms, although often initially pursued in one or two lead- ing cities captured by DB, are frequently extended to subnational levels. India developed its own subnational indicator system inspired by DB to encourage and reward reform competition among its states to make it easier and quicker for businesses to operate. From 2014, the government of India 20 introduced the Business Reform Action Plan, overseen by the Department of Industrial Policy and Promotion. The Russian government developed key performance indicators inspired by DB, then introduced a subnational program of key performance indicators to encourage virtuous competition among provincial governors. The program was coordinated by a national nongovernmental organization, the Agency for Strategic Initiatives, which also generated standards to guide regional policy makers on how to improve the regional business environment and attract investment. In Morocco, the measured progress of DB reforms at the national level, combined with a na- tional strategy of decentralization, spawned support for subnational indica- tors and reforms extending to the Marrakesh region and beyond. Several country case studies revealed serious limitations to the DB indica- tors and agenda in capturing business environment reform priorities. In no case study did IEG find that the problem areas identified by DB were directly inconsistent with national development plans and objectives, but in many cases, they were not development priorities. In the cases of fragility, conflict, and violence countries, such as Afghanistan and the Democratic Republic of Congo, political stability and risk of conflict weighed heavily on business- es and constrained any potential supply response to regulatory reforms. In many other countries, including Morocco and Jordan, the lack of a level playing field for domestic SMEs meant that DB failed to capture the substan- tial disadvantages imposed on SMEs through advantages granted to favored Independent Evaluation Group World Bank Group    21 foreign or domestic investors through tax advantages, access to special economic zones, preferential access to land or credit, or explicit subsidies. In several countries, state ownership also limited private sector opportunity in multiple sectors. In multiple countries, the presence of corruption, large informal sectors, or state capture weakened the relevance of DB indicators to binding business constraints. For example, in one country, despite a thor- ough streamlining of dealing with construction permits (DWCP), businesses reported that, at the end of the process, approval still required a meeting with an official who expected a bribe. The relevance of the DB agenda is weaker in countries where structural or institutional factors act as binding constraints. For example, Rwanda could not address the disadvantages of being small or landlocked through DB re- forms. Further, its 2019 CPSD pointed to a host of binding constraints more pressing than the DB agenda, including low skills (human capital), limited access to and high cost of energy, high transport and information and com- munication technology costs, restricted access to land, and an unlevel playing field for competition. In India, critical aspects of binding constraints cited by stakeholders or revealed in enterprise surveys lay outside the DB agenda, ranging from weak infrastructure to land and labor constraints. In Jordan, key constraints included massive unemployment and poverty, as well as income disparity and gender inequality. In China, a host of issues ranging from inef- ficient bureaucracy to bank debt figured among leading constraints cited by stakeholders and experts interviewed but mostly missed by the DB agenda. In some countries, it is not clear whether the domestic SME focus of the DB The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Chapter 2 agenda was the real focus of the reform efforts. In Jordan and Afghanistan, the reforms were being pursued largely with an audience of donors and potential foreign investors in mind. With some indicators, like protecting minority investors, it is not clear how well the base case assumptions map to the typical domestic SME, although the indicator may well resonate with a foreign investor. Rwanda pursued an improved external reputation to attract foreign investment, in part through the publicity of being a recurrent top reformer. Russia sought to burnish its international image while also aiming to improve the domestic business environment. Although appealing to addi- tional audiences—whether domestic voters, foreign donors, or international investors—may be considered a benefit of DB reforms, it can also inspire strategic behavior, pursuing reforms that move indicators without yielding tangible benefits to domestic SMEs. How client countries use the indicators depends on their motivation and their capacity. Over time, the DB agenda can lose its relevance for several reasons, when (i) non-DB constraints become binding after early DB reforms, (ii) pending DB reforms prove less tractable, and (iii) a DB indicator does not adapt to chang- es in the underlying business process or technology. First, over time, active reformers may have addressed the most pressing constraints measured by DB and move on to other policy priorities. This may explain why top 10 EoDB countries like New Zealand, Denmark, and the United States show a rate of one or fewer reforms per year from FY10 to FY20. Second, within each policy area, as the more tractable areas are addressed, the remaining DB agenda 22 may rest with longer-term or politically more difficult reforms. India, like many other countries, progressed first on relatively easy procedural simplifi- cation in starting a business, paying taxes, trading across borders, and DWCP, which reduced the time and the costs involved. Reforms in more difficult areas involve legal processes, such as resolving insolvency, registering prop- erty, and enforcing contracts. Indicative of the long-term nature of these challenges, despite important reforms to improve its insolvency framework, business associations reported the persistence of slow, cumbersome, and inefficient resolution in court. Despite serious effort to improve its insolven- cy framework, business associations reported the persistence of slow, cum- bersome, and inefficient resolution in court. Third, where DB indicators are not adapted, the reform agenda may shift away from them as the underlying technology or nature of the business area evolves. For example, progress in e-government may invalidate traditional measurements of number of proce- dures or assumptions about the time taken by each step in such contexts as starting a business and paying taxes. In getting credit, advances in use of big data and the emergence of digital financial services can reduce the relevance of existing credit information indicators. Business Area Relevance of Doing Business Indicators The overall EoDB score serves as a general index of the overall regulatory Independent Evaluation Group World Bank Group    23 environment. A low EoDB score corresponds generally to other indicators of administrative burden or weak regulatory quality. For example, statistical analysis shows a high correlation (77 percent) between the EoDB score and the World Economic Forum’s survey-based indicator of the burden of public administrative requirements perceived by business managers (figure 2.2). The ordering of reform priorities using DB indicators and business enterprise surveys shows some alignment. Within the areas DB measures, IEG consid- ered statistical evidence comparing the ordering of priorities indicated by DB scores to the ordering of priorities indicated by businesses responses to enterprise surveys. The IEG analysis matched four overlapping categories between DB and enterprise survey responses by country and year, namely tax administration, trade regulations, access to electricity, and access to finance. Based on 95 observations, the analysis found a perfect match in the ordering of priorities in 28 percent of cases, a close match (with only one single posi- tion difference in rank) in 41 percent of cases, and a mismatch of rankings in the remaining 31 percent of ratings. Differences in methodology and among the firms sampled in surveys versus those described in the DB base case assumptions may explain some of this deviation, but it points to the value of using multiple sources of evidence to guide reform priorities.  orrelation between the Doing Business Ease of Doing Figure 2.2. C Business Overall Score and the World Economic Forum Burden of Regulation Indicator The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Chapter 2 Source: Independent Evaluation Group, statistical analysis of Doing Business indicators. Note: DB = Doing Business; WEF = World Economic Forum. DB indicators do not capture well the real conditions experienced by busi- nesses within each area covered by an indicator.4 IEG notes a low correlation between DB scores and how firms operating within countries report their experience. Regarding the DWCP indicator, there is only a 23 percent cor- relation between the DB measure of time to get a construction permit and the experience reported by firms in surveys. For the getting an electricity connection indicator, this correlation is only 10 percent. For both indica- tors, the DB indicators tend to overestimate time compared with actual firm experience. Regarding the time to import indicator, there is only a 42 per- cent correlation with firm experiences reported in surveys despite improved 24 methodology since DB2016, an underestimate. Regarding the time to export indicator, the correlation of DB to enterprise survey responses is 35 percent, with DB again underestimating relative to survey responses.  orrelations and Over- or Underestimating DB versus Table 2.1. C Enterprise Surveys Corresponding Correlation Over or DB Indicator ES Indicator Coefficient (%) Underestimate Time to get a construc- Time to get a con- 23 Overestimate tion permit struction permit Time to import (DB16–20 Time to import 42 Underestimate methodology) Time to export (DB16–20 Time to export 35 Underestimate methodology) Time to import (DB06–15 Time to import 25 Overestimate methodology) Time to export (DB06–15 Time to export 23 Overestimate methodology) Days to get electricity Days to obtain 10 Overestimate connection electrical connec- tion Source: Independent Evaluation Group, statistical analysis of DB and enterprise survey data. Note: DB = Doing Business; ES = enterprise survey, various years–DB data matched to timing of enter- prise surveys.. Within each indicator area, there are limitations to how the indicator applies Independent Evaluation Group World Bank Group    25 to conditions faced by businesses in the field.5 IEG conducted five deep dive studies to review the most reformed indicator areas. They show strengths and limitations of the relevance of each indicator set to the business area it covers. The dives considered information from several sources: two liter- ature reviews, a portfolio review, country case studies, and interviews with experts and practitioners. The latter give DB substantial credit for drawing attention to administrative burdens on firms and launching discussions about its individual policy areas. Limitations in DB indicators are generally attributed to shortcomings in their defined topical coverage and in the representativeness of the base case scenario.6 Given its challenging task of collecting data for 190 countries every year, DB has stylized or simplified its coverage of certain areas. This involves the use of base case scenarios, intended to enhance comparability between countries, and coverage of only one or two cities. The reliance on intermediaries (for example, lawyers, accountants, freight forwarders) great- ly streamlines data collection relative to surveys. Yet these measures can constrain the representativeness of reported data. Lack of substantive coverage can limit the extent to which the indicators can (or should) guide reforms or reflect reform progress. DB reports caution readers on a few of the limits to the indicators (box 2.6). One issue is the detail and granularity of indicators, which are by necessity simplified. DB can be a crude instrument for monitoring reform measures, often crediting reform on issuance of a law or regulation before implementation, or failing The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Chapter 2 to recognize certain important reforms. Questions on granularity and com- prehensiveness raise the concern that indicators as reform metrics can leave important things out in individual areas. For example, a World Bank expert on insolvency noted that, although the indicator had improved to cover 13 or 14 legal subindicators, clients really would need at least 25 to have the granularity needed for understanding and guidance on the issue. The characteristics describing the DB base case scenario are not always con- sistent with those experienced by the typical domestic SME. To understand relevance issues more clearly, IEG conducted detailed “deep dives” to ex- amine five DB indicators that were the most popular areas of reform. These deep dives found inconsistencies between specifications of the base case scenario and the typical domestic SME. One challenge is that the scenario behind some indicators describing a prototypical firm or transaction could not be the same one used for the scenarios underlying other indicators. For example, the firm for starting a business must not engage in international trade, so would never encounter a trading across borders situation. That same firm (with five partners) would not qualify for the scenario in protect- ing minority investors. The findings on relevance from these deep dives are summarized in table 2.2. More details can be found in appendix D. 26 Box 2.6. Caveats in Doing Business Reports, 2010–20 The Independent Evaluation Group found 87 caveats in Doing Business reports alerting readers about some limitations of the indicators. Three types dominated: the limited scope of the indicators, the limited extent to which indicators capture the full range of legal and regulatory priorities, and the limited ability of indicators to capture coun- try-specific conditions (figure B2.6.1). Figure B2.6.1. Caveats in Doing Business Reports Other Indicators cannot fully 3% capture country-specific conditions 14% 24% Indicators do not capture the full range of legal and regulatory priorities 59% The scope/focus of the indicators is limited Source: Independent Evaluation Group, analysis of Doing Business 2010–20 reports (World Bank 2009, 2010, 2011, 2012, 2013, 2014a, 2015a, 2016c, 2017b, 2018b, 2019a), and supervised machine learning. Independent Evaluation Group World Bank Group    27  ey Findings on Relevance from Doing Business Indicator Table 2.2. K Deep Dives DB Indicator Summary Findings Starting a business » A strong motivator for reform of business registration, the indicator doesn’t capture key elements of the framework for start-up of new businesses. » The indicator does not capture qualitative features such as: (i) whether the legal framework allows for a wide variety of legal forms appropriate to the nature of the business; (ii) some important qualities of the business registry itself—for example, is it electronic, backed up, up to date (and free from “ghost” companies), and populated with full information on owners and directors (including gender), the structure of businesses, The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Chapter 2 and their age; (iii) the clarity of the framework of laws and reg- ulations for business operation and governance. » The base case scenario does not describe a typical firm con- sidering formalization, given characteristics specified such as having five married partners all of the same gender. Getting credit » The indicator measures two key elements of a country’s finan- cial infrastructure: the legal rights of borrowers and lenders, and the availability of credit information about firms and individuals. » Within its area of coverage, experts point to the limited scope of the indicator relative to what is covered by guidance from the United Nations Commission on International Trade Law on secured transactions. » The indicator does not capture a host of other factors (includ- ing imperfections in the banking and financial sector) that can impede the supply of credit to businesses. Trading across bor- » The indicator benefits from broad coverage and comparabil- ders ity across countries as well as annual reporting. It is useful in pointing to cases where trade facilitation is problematic (World Bank 2019b). » Client countries find the indicator motivating, often using it as a goal, and find the subindicators generally concrete and thus easy to understand. » The indicator measures only compliance costs, ignoring whether trade regulations are achieving their policy objectives. » The base case scenario focuses on only one good (export or import) going from one point of origin to one destination. This makes the indicator highly sensitive to changes affecting the specific case, but not necessarily representative as a national indicator. (continued) 28 DB Indicator Summary Findings Dealing with con- » The indicator is seen as one of the most complex DB measures struction permits since many agencies from different government levels get involved. » The current indicator reflects a reform in DB2016 that added a building quality control index (composed of six subindexes). » There remain concerns about the indicator’s inattention to is- sues of fire safety, use of nonconventional (potentially hazard- ous) materials for construction, country geological conditions, and informal construction. » The specifications of the base case scenario are so limiting that they may not apply to many real situations in the field. Paying taxes » The indicator has several components: a “tax payments” sub- indicator; a “time” subindicator that estimates the time it takes a firm to “prepare, file, and pay” all of its taxes; a “total tax and contribution rate” subindicator that tracks the amount of taxes and mandatory contributions borne by the business; and the “post-filing index” subindicator (added in DB2017), including time to comply with several requirements (for example, time to complete VAT refund collection). » Experts find that the administrative component provides a simple and complete account of the administrative burden on firms. Although it is not detailed enough to guide the design of reforms, it is helpful in drawing attention to the compliance costs of taxes. » Many experts find the “tax policy” side of the indicator focusing on tax rates deviates from the focus on business compliance costs into public policy. » The subindicator on “total tax and contribution rate” is seen Independent Evaluation Group World Bank Group    29 as a weak guide to policy because it rewards lower tax rates, lacking any basis in fiscal, economic, or social policy and any adaptation to different national policy priorities. » Overall, the primary value of the “paying taxes” indicator is seen as calling attention to tax compliance burden, but its utility to guide reforms is seen as extremely limited. » Nonetheless, discussions with country officials indicate that dialogue on paying taxes can be valuable and expose and connect them to good practices in reforming countries. Source: IEG Deep Dives – see appendix D.. Note: DB = Doing Business; VAT = value-added tax. World Bank Group Country Strategy and Operations The Bank Group uses DB in country strategies to identify reform needs and to motivate future operations in priority areas. Bank Group expert practi- tioners find near universal appreciation of the relevance and value of DB indicators to initiate reform dialogue. Most responses favored the relevance of the indicators to country priorities. Yet there was an almost even division of positive and negative responses on the value of the indicators to correct- ly identify binding constraints. And the preponderance of views weighed against the use of DB indicators to monitor and evaluate reforms. The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Chapter 2 Many World Bank country strategies make substantial references to DB or propose a DB-related work program or both. IEG examined 61 country strategy documents with corresponding IEG reviews and found that 64 per- cent of the strategies substantially referenced DB or proposed a DB-related work program. These strategies forecast planned interventions including improving business laws and regulations (38 percent), streamlining proce- dures (34 percent), using electronic or automated systems (15 percent), and conducting diagnostics (11 percent). Examples include policy development and regulatory streamlining in the Philippines (World Bank 2019d); policy dialogue and advice in Mexico (World Bank 2018d); and policy and project support in Rwanda (World Bank 2019c). The World Bank’s own assessment of the quality of countries’ policy and institutional frameworks uses DB indica- tors as inputs to inform its coverage of the business regulatory environment (starting a business, resolving insolvency, registering property, protecting minority investors [shareholder rights], and DWCP); nontariff trade mea- sures (trading across borders); property rights (enforcing contracts); and efficiency of revenue mobilization (paying taxes). DB is the most popular source of business environment information but not the only one, even among Bank Group products. IEG reviewed 18 CPSDs, the Bank Group’s current comprehensive private sector development analytics produced jointly by IFC and the World Bank. It found that the CPSDs in- corporate DB indicators as a leading input in a way that can be quite inde- pendent from and integrated with information from other Bank Group and 30 global indicators and evidence sets (figure 2.3). IEG’s review of 50 evaluated IFC AS projects found that many use other primary indicators, including some standard AS indicators designed to produce data more focused on the scope, depth, and timing of the engagement, and more granular and aligned with project objectives. ndicators Used as Measures of Business Environment Figure 2.3. I Constraints in Country Private Sector Diagnostics DB indicators (n = 16) 89% World Bank enterprise 89% survey (n = 16) WEF GCI Global (n = 14) 78% Indicator World Bank Logistics Performance Index (n = 13) 72% Bertelsmann Stiftung Transformation Index (n = 5) 28% IMF country report (n = 4) 22% 0 20 40 60 80 100 Percent Source: Independent Evaluation Group. Note: DB = Doing Business; GCI = Global Competitiveness Index; IMF = International Monetary Fund; WEF = World Economic Forum. DB informs a substantial share of the Bank Group’s projects providing fi- Independent Evaluation Group World Bank Group    31 nancing, advice, and technical assistance to client countries on the business environment. This DB-informed portfolio consists of 676 projects represent- ing $15.5 billion in commitments during FY10–20 (table 2.3). It includes projects that (i) use DB in their Board documents to justify the project, or (ii) have one or more DB indicators in either their objectives, or (iii) have one or more DB indicators as monitoring indicators, or (iv) are intended specifically to inform DB indicators. The DB-informed portfolio constitutes approximately 28 percent of the 2,445 projects dealing with the business environment and approximately 3 percent of 22,761 Bank Group projects (figure 2.4).  ummary of Doing Business–Informed Portfolio, Approved in Table 2.3. S Fiscal Years 2010–20 (projection) Projects Interventions Commitments (US$, Institution (no.) (%) (no.) (%) millions) (%) World Bank lending 269 40 517 43 14,853 96 Subtotal 269 40 517 43 14,853 96 World Bank ASA 165 24 173 14 379 2 World Bank RAS 58 9 81 7 20 0 IFC AS 184 27 428 36 287 2 Subtotal 407 60 682 57 686 4 The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Chapter 2 Total 676 100 1,199 100 15,539 100 Source: Independent Evaluation Group, portfolio review analysis assisted by supervised machine learning. Note: Projected based on population and sample sizes. Volume/commitment/funds managed were identified or estimated according to what was allocated to Doing Business–related interventions. If not explicitly stated, it was estimated based on the number of components, subcomponents, or activities. Consultations with World Bank Group resulted in exclusion of some projects. ASA = advisory services and analytics; IFC AS = International Finance Corporation advisory services; RAS = reimbursable advisory services.  hare of Doing Business–Informed Projects in World Bank Figure 2.4. S Group Portfolio, Fiscal Years 2010–20 Rest of business environment 1,769 8% Doing Business 3% 676 89% Remainder of portfolio 20,316 Source: Independent Evaluation Group, portfolio review.. The World Bank provided lending support that is informed by DB to 97 coun- tries, and the World Bank and IFC together provided advisory services and 32 analytics informed by DB to 126 countries. In total, 136 countries were sup- ported by DB-informed operations. By income level, 29 percent of projects were delivered to low-income countries, 38 percent to lower-middle-income countries, 25 percent to upper-middle-income countries, and 8 percent to high-income countries. Reimbursable advisory services figured prominently in services delivered to high- and upper-middle-income countries. Regional- ly, Sub-Saharan Africa was the region with the highest number of approved projects (34 percent), while Middle East and North Africa had the highest average in lending projects per country (4.5), and South Asia had the highest average per country in advisory projects (4.8). Regional projects, mostly in Sub-Saharan Africa and Latin America and the Caribbean, constituted 7 per- cent of the estimated portfolio. Within the DB-informed portfolio, besides general support to improve the EoDB (27 percent), the most popular reform interventions focused on a selection of business areas covered by the indicators. They include trading across borders (28 percent), starting a business (25 percent), getting cred- it (18 percent), and DWCP (18 percent), with paying taxes and registering property close behind (figure 2.5). Of the identified projects, 47 percent use DB indicators as justification, 29 percent use them to measure reform prog- ress, and 15 percent use them in project objectives. Tanzania’s First Business Environment for Jobs development policy operation (FY16) exemplified DB indicators as objectives, aiming to reduce business start-up time from 26 to 10 days and procedures from 9 to 3 days.  Independent Evaluation Group World Bank Group    33 Project objectives in the DB-informed portfolio focused most often on improving a law or regulation (27 percent), reengineering a process (16 per- cent), building capacity and training (15 percent), and conducting a di- agnostic (13 percent). Turkey’s Second Restoring Equitable Growth and Employment Programmatic Development Policy Loan (P123073; FY11), for example, helped government to adopt a new commercial code enhancing protection of minority shareholders. The IFC AS Georgia Investment Climate Project (599537; FY14) assisted Georgia to streamline regulations and pro- cedures (including through e-government) related to trade, enforcement of contracts, and insolvency. The IFC AS Afghanistan Business Enabling Project (602848; FY18) provided training and capacity building to Afghan agencies including peer-to-peer learning.  rojected Doing Business–Informed Portfolio FY10–20 by Figure 2.5. P Indicator Area and World Bank Group Institution (number and percent of DB-informed projects) Ease of doing business 27% Trading across borders 28% Starting a business 25% Getting credit 18% Dealing with construction permits 18% Paying taxes 16% Registering property 16% Enforcing contracts 13% Resolving insolvency 11% 9% The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Chapter 2 Getting electricity Protecting minority investors 7% Contracting with the government 1% Employing workers 1% 0 50 100 150 200 Projects (no.) World Bank lending World Bank RAS World Bank ASA IFC-AS Source: Independent Evaluation Group, portfolio review analysis. Note: Projected based on population and sample sizes. Results were multiplied by a factor of 226/107 for World Bank lending projects, and 789/452 for World Bank ASA projects. Where a project or component specified an indicator it would work on, it was included under that indicator. Projects are counted more than once if supporting more than one business area. Projects with components that do not identify a specific indicator or that discuss reform in multiple indicator areas without a specific allocation of project resources to each were categorized as “ease of doing business.” ASA = advisory services and analytics; IFC-AS = International Finance Corporation advisory services; RAS= reimburs- able advisory services. DB most often informs projects as a project rationale (48 percent of proj- ects) but also serves as a project indicator (29 percent) or project objective (15 percent; appendix A). Given that some projects use DB in more than one way, overall 42 percent (287 of 676) of projects use it as either an indicator or objective or both. The use of DB as a project indicator declined somewhat between FY00–05 and FY06–10, whereas the use of DB as a project objective slightly increased over the same period. A few projects seek to generate or inform DB indicators (13 percent). 34 Research Academics and other researchers have found great utility in having time series of DB indicator data updated annually, covering most countries in the world in the areas of regulatory policy covered by the indicators, and it is fertile subject matter for research. The 2018 external audit concludes that the ease of doing business indicators “are one of the World Bank’s most im- portant contributions to research and public policy” (Morck and Shou 2018, 3). DB reports that think tanks and research organizations use the indicators “both for the development of new indexes and to produce research papers” regarding the relationship of business regulation to economic outcomes (World Bank 2016c, 21). Business indicators inform or are used in a large amount of research. In several areas, there are no comparable annually up- dated data available for so many countries. The evaluation team found an extensive body of research literature that uses or focuses on DB indicators and the business areas they track. The DB team shared a database of more than 400 articles of relevance to DB from 100 leading journals. IEG’s SLR found close to 1,900 articles of potential relevance based on search terms and a review of abstracts in 10 DB indicator areas. DB2019 reported “more than 3,400 research articles discussing how regulation in the areas measured by Doing Business influence[s] economic outcomes” published in peer-reviewed academic journals, 1,360 of those published in Independent Evaluation Group World Bank Group    35 the top 100 journals, and another 9,450 “published as working papers, books, reports, dissertations or research notes” (World Bank 2018b, 32). Doshi, Kel- ley, and Simmons (2019, 30) point to a significant literature in “critical legal research as well as statistical studies” critiquing the validity of DB indicators, identifying “methodological, substantive and conceptual problems with rely- ing on the EoDB indicators for assessing the business environment.” A review of DB’s own database from 100 top academic journals indicates that research has concentrated in a few areas, with disproportionate attention to starting a business, trading across borders, and protecting minority investors. IEG’s SLR shows a concentration of rigorous articles about starting a business and trading across borders, with resolving insolvency and getting credit also proving popular. Conversely, getting electricity and registering property have not been widely treated in rigorous studies of outcomes (appendix F). Doing Business Indicators over Time The DB indicators have evolved, including the introduction of new indicators and revisions to the methodology of existing ones. Indicator changes are necessary to reflect learning and evidence about their relevance and effec- tiveness. Given the many limitations noted above, many experts would like to see further changes. A series of changes from DB2015 through DB2017 updated all of the indicators, and many of the updates won praise from sub- ject experts. For example, tax experts appreciated the addition of postfiling requirements. Experts on construction regulation appreciated the addition of a building quality index. IEG acknowledged the validity of eliminating the documents subindicators from trading across borders (World Bank 2019b). In The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Chapter 2 addition, IEG recommended the expansion of indicators to include unrepre- sented elements of the regulatory environment, such as environmental and competition regulation (World Bank 2015b). Evidence from this evaluation suggests that additional modifications should be incorporated into future revisions of the indicators to enhance their rel- evance to country reforms. These modifications can refine the indicators to (i) better capture legal and regulatory attributes with developmental conse- quence, (ii) better attune to conditions experienced by local businesses, and (iii) adapt to procedural or technological innovations or evolution that alter the nature of what is being measured. The complementary work of DEC’s expert panel can help inform this process. Each DB indicator change is not without cost, so it matters how such chang- es are introduced. Academic users can be among the most affected when val- ued time series are disrupted by changes in methodology. DB can limit (and sometimes has limited) this effect by making available indicators using the old methodology for a period after transitions. Client countries are affected when progress toward targets rooted in one methodology are not rewarded under a new one. Any disruption in the continuity of time series data can affect both client and project targets based on former indicator construction. Evidence suggests that distress increases if client counterparts do not feel consulted or forewarned. If such changes are infrequent and well commu- nicated, they can serve to enhance the value of the DB indicator set. Not all 36 observers agree, and the 2018 external audit recommended that the World Bank “minimize methodology changes except to fix confirmed problems with existing methodology” (Morck and Shue 2018, 2). Independent Evaluation Group World Bank Group    37 1  The Organisation for Economic Co-operation and Development DAC Network on Develop- ment Evaluation defines the evaluation criterion relevance as the answer to the question, “Is the intervention [or program or policy] doing the right thing?” To assess it, evaluators explore “how clearly an intervention’s goals and implementation are aligned with beneficiary and stakeholder needs, and the priorities underpinning the intervention. It investigates if target stakeholders view the intervention as useful and valuable” (OECD 2021). 2  As noted in the “influence model” theory of change, a low country ranking may come to the attention of leadership through multiple channels, including donors, foreign or domestic investors, the press, or civil society. 3  The Omnibus Law, discussed in appendix C, had some limitations, including the weakening of environmental screening. The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Chapter 2 4  An additional difference with enterprise surveys is geographic coverage—typical enterprise surveys cover more cities or regions than does DB. 5  Chapter 3 addresses the extent of evidence on individual indicators’ links to outcomes. 6  This evaluation does not attempt to generate advice on the design and methodology of in- dividual indicators. That is being explored in greater detail in parallel work commissioned by the World Bank’s Development Economics Vice Presidency. 38 3 | The Effectiveness of Doing Business: Is It “Doing Things Right”? Highlights Doing Business (DB) reports make many claims linking reforms measured by its indicators to outcomes, such as job creation and economic growth. Only a minority of these claims can be con- firmed through articles published in leading journals, and a small- er share have been replicated or confirmed through a rigorous methodology. Weakly evidenced claims can create a reputational risk for the World Bank Group and its Development Economics Vice Presidency. Although many countries and Bank Group projects use movement in DB indicators as an outcome measure, such movements are in- consistently linked either to reform implementation or to economic outcomes, such as increased investment, employment, or produc- tivity. The Independent Evaluation Group’s country case studies show both strong movement of DB indicators and tenuous links to measurable development outcomes. Three-quarters of evaluated country strategies with DB objectives mostly or wholly achieved those objectives, but only 45 percent also showed improvements in DB indicators. In some countries, impact was limited by a failure to address binding constraints. The Bank Group DB-informed project portfolio is generally suc- cessful in achieving project objectives but less so in three areas that require deep institutional reforms (enforcing contracts, reg- istering property, and resolving insolvency). Although client com- mitment and capacity are important, most factors of success—in-   39 cluding complementary analytic work, client engagement and follow-up, effective coordination, and monitoring and evaluation— are largely within the Bank Group’s control. Bank Group project experience shows that effective practices include a focus on binding constraints; use of a strong interagency coordination unit; timely availability of expertise; long-term, com- prehensive engagements; public-private dialogue; client capacity building and knowledge sharing; and peer-to-peer learning.   40 This chapter examines the extent to which activities informed by DB are yielding intended objectives and results—their immediate and interme- diate outcomes. This evaluation is ultimately concerned with the devel- opment effectiveness of how DB is used by client countries and the Bank Group—the extent to which program, policy, and project reforms achieve their objectives and results, systemically and for specific beneficiary groups. A key question concerns how much benefit an improvement measured by DB yields in terms of actual results and development outcomes such as in- creased growth, investment, business entry, productivity, or employment. Research Evidence IEG’s desk-based review of literature identified by the DB team finds sig- nificant associations of what DB indicators measure with outcomes but does not establish causation. The desk review took stock of evidence on DB effectiveness based on a database of 426 articles from 100 leading journals. The database was organized by indicator. Some areas had abundant coverage (starting a business, protecting minority investors, and trading across bor- ders), but many others had much lighter coverage (for example, DWCP and getting electricity). The desk review did not assess the methodological rigor of each article but did consider whether the article shed light on the rela- tionship between what a DB indicator measures and an outcome, thus lim- iting its reporting to 75 articles of strong relevance. Findings confirmed by Independent Evaluation Group World Bank Group    41 multiple articles were considered to have “strong evidence” (box 3.1). Where the findings of two or more articles contradicted each other, the evidence was classified as “mixed.” On this basis, very few findings were confirmed by multiple articles. IEG conducted a second rigorous SLR, which found that although there is a reasonable depth of coverage of some indicators, others have sparse ev- idence causally linking outcomes to reforms tracked by DB. To capture a broader body of literature and understand rigorously established relation- ships between DB-related action and outcome, the SLR followed method- ological criteria consistent with norms established by leading practitioners of systematic reviews (appendix F) to identify studies in English meeting its inclusion criteria for populations, interventions, comparison groups, out- comes, and study designs. The SLR cast its net broadly, initially identifying 9,221 studies from multiple data sources that pertained to the 10 indexed DB indicator areas. After applying the filters for inclusion, the SLR identified 103 studies of relevant reforms meeting its criteria, including that allocation to intervention and control groups was random, and selection bias had been addressed by design (figure 3.1).  our Doing Business Outcome Findings Evidenced by Multiple Box 3.1. F Articles Published in Leading Journals Starting a business: Higher entry costs or more steps or documents are associated with less firm creation, growth, or profitability. The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Chapter 3 Getting credit: Stronger legal rights are associated with more lending, more financial activity, or lower interest rates. Paying taxes: Higher tax rates constrain entrepreneurship (rate of business creation) and investment rate. Trading across borders: Better trade facilitation as measured by Doing Business shows a correlation with increased trade flows. Source: Independent Evaluation Group, desk review of Doing Business literature database.. The SLR confirms several of the relationships between the laws or regula- tions tracked by DB and significant development outcomes, while also rais- ing some cautions and identifying gaps. In general, research findings do not necessarily use DB’s definition or base case to measure critical phenomena. For example, there is evidence that low business entry costs encourage firm entry or formalization and growth, but not all findings are connected to DB’s specific formulation of entry. Sparse evidence confirms that the accessibility of land services and the process of transferring property is positively related to increased activity in commercial rental and property markets and access to credit. Mixed evidence generally confirms (with considerable nuance) a relationship of improved credit information and more expansive collateral laws to increased credit to private firms. Evidence supports the relationship of tax administration reforms to enhanced tax compliance and reduced busi- 42 ness perceptions of tax administration as a constraint. Evidence supports a positive relationship between trade facilitation reforms and trade flows. Ev- idence on contract enforcement supports a general relationship of improved contract enforcement and judicial efficiency to broader outcomes such as investment, productivity, and profits; yet this evidence does not link to the specific good practices advocated in DB. Evidence on insolvency supports the idea that improvements in the efficiency of insolvency can lower the cost of borrowing and enhance private firm access to credit.  igorous Studies of Outcomes Identified by Structured Figure 3.1. R Literature Review Relating to Indexed Doing Business Indicators 25 Articles meeting inclusion criteria (%) 20 3 5 15 5 18 1 1 10 15 12 5 10 10 8 7 5 0 3 0 Starting Construction Getting Registering Getting Protecting Paying Trading Enforcing Resolving a business permits electricity property credit minority taxes across contracts insolvency investors borders Opening Getting Financing Day-to-day Secure business a business a location operations environment Independent Evaluation Group World Bank Group    43 Indicator Studies reporting positive effects Total studies (no.) Source: Independent Evaluation Group, structured literature review. The SLR did not find empirical evidence supporting outcomes linked to the DB measures of DWCP (construction permits) or of getting electricity, and it raises some cautions about unintended consequences of reforms encouraged by DB. The cautions arise from findings in the literature on the potential for (i) gender and environmental consequences of simplified business entry; (ii) “getting credit” reforms to increase the exclusion of some borrowers; (iii) protection of minority shareholder rights to increase the cost of equity, debt, and audit fees; (iv) efficiency/ quality trade-offs in judicial efficiency; and (v) insolvency reforms to promote capital intensity with potential labor market effects. On trade facilitation, there is a lack of robust evidence about the im- plications of reforms for other public objectives (for example, public health, safety, the environment, and reducing informality). Effectiveness Claims Made in Doing Business Reports DB reports frequently associate the DB reform agenda with the Bank Group’s institutional twin goals of ending extreme poverty and promoting shared prosperity. The association of DB reforms with reduced poverty and en- The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Chapter 3 hanced prosperity can be found in even the earliest DB report but was quite explicit in DB2019, in which then–World Bank president Jim Young Kim stated: “The reforms that the report inspires will help people reach their aspirations; drive inclusive, sustainable economic growth; and bring us one step closer to ending poverty on the face of the earth” (World Bank 2018b, v). Earlier, in DB2017, the World Bank’s then chief economist stated, in relation to the twin goals, “Doing Business helps us make progress on one crucial strategy for meeting these goals—offering market opportunities to everyone” (World Bank 2016c, vii). DB2010 (and others) linked its measured reforms with the ability of people with lower incomes to find jobs and escape poverty. IEG finds that DB reports make claims that improving the legal and regu- latory conditions for businesses as summarized by DB’s general index will benefit a variety of development outcomes. To better understand claims for explicit links of reforms tracked by DB with development outcomes, IEG used supervised machine learning and human review to identify claims about out- comes in DB reports issued from 2010 to 2020. This review identified 89 af- firmative claims linking improvements in what the DB indicators measure to better development outcomes (or, conversely, worse DB-tracked conditions with worse development outcomes). Of these, 23 (26 percent) concerned the general association of improvements in DB areas (including the overarching EoDB indicator and distance to frontier) with improvements in outcomes. Like the claims in the previous paragraph, such claims do not map a reform path, but focus instead on outcome claims. 44 Of 66 remaining claims pertaining to a specific indicator area of DB, most (51 percent) did not reference the immediate outcome of the claim, leaving the causal links to the described outcome or impact undefined. Of those that did define an immediate outcome, they were most commonly in the cate- gories of reducing the cost of doing business, followed by streamlining of procedures and reducing days to complete procedures. The overwhelming majority of all the 89 claims identified (68) did refer to intermediate out- comes, led by an increase in business entry (33 percent of claims), an in- crease in investment (30 percent), or an increase in formality (20 percent). IEG also identified within the 89 claims 97 asserted links associating DB- related reforms with identifiable development impacts, led by job creation and economic growth (figure 3.2).1 As an example, DB2015 states that “research provides strong evidence that reforms making it easier to start a business are associated with more firm creation, which in turn is strongly associated with job creation and economic growth” (World Bank 2014a, 33). Impact claims of increased job creation were most commonly associated with intermediate outcome claims to increased business entry, formality, and investment. Impact claims of economic growth were mostly linked to intermediate outcome claims regarding increased business entry, with a minority also related to claims about increased formality and investment. Although some of the claims contained in DB reports meet a high standard of evidence, most claims do not, which can create a reputational risk to the Independent Evaluation Group World Bank Group    45 Bank Group. IEG found that only 13 percent of articles cited as evidence used robust methods and study designs meeting the criteria of the SLR, and 10 percent of results cited had been replicated in multiple articles in DB’s own literature database.2 If claims for evidence of outcome made in the report are not held to a consistent and high standard, or if they are cited selectively only to support the case for reform, the risk is that the DB report, DEC, and the Bank Group may be increasingly regarded as advocates more than as trustworthy interpreters of evidence (box 3.2). Although the stan- dards need not be as rigorous as those used in IEG’s SLR, the principle is that transparent and systematic reporting of the available evidence with atten- tion to nuance and complexity is necessary to guard against potential bias, oversimplification, overgeneralization, and reputational risk.  laims Linking Doing Business–Related Reforms to Figure 3.2. C Development Impacts Job creation (n = 43) 36% Growth (n = 33) 27% Development impact Poverty alleviation (n = 9) 7% Increased productivity (n = 4) 3% Shared prosperity (n = 2) 2% Other (n = 6) 5% The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Chapter 3 Not clear (n = 24) 20% 0 5 10 15 20 25 30 35 40 Number of claims (percent) Source: Independent Evaluation Group, analysis of DB reports 2010–20. Note: Bar labels denote percent of all claims..  omparing the Doing Business 2020 Literature Review with the Box 3.2. C Independent Evaluation Group Structured Literature Review The second chapter of Doing Business (DB) 2020 presents a literature review citing 53 articles that together cover all DB areas and were published between 2013 and 2019. Of the articles, 90.5 percent supported the arguments of DB by confirming the relevance of the DB reforms’ agenda or finding that positive results associated with specific undertaken reforms were consistent with what DB indicators reward. (The Independent Evaluation Group’s structured literature review [SLR] found that 85 per- cent of rigorous articles are associated with positive benefits of reforms in terms of an economic or development outcome.) Nevertheless, these articles mostly did not meet the criteria of the Independent Evaluation Group’s SLR. Of 35 articles overlapping with the coverage of the SLR, only 4 were included. These four related to the relationship of customs delays to export volume, of contract enforcement to entrepreneurship among individuals with higher levels of education, of credit information to loan de- faults, and of the improved legal rights of creditors to lending activity and credit terms. Source: Independent Evaluation Group, desk review of Doing Business 2020 (World Bank 2019a). 46 Country Reforms Although many countries, donors, and Bank Group projects use movement in DB indicators as an outcome measure, such movements are inconsistently linked both to reform implementation and to economic outcomes. A review of IEG’s 10 case studies reveals strong movement of DB indicators in most countries but tenuous links to measurable development outcomes. Some countries show little increase in investment, employment, or productivity. Overall, IEG finds a mixed picture of indicators’ links to country outcomes (appendix C): » In Morocco, despite both an improvement in ranking from 128th of 183 countries in EoDB in 2010 to 53rd of 190 countries in EoDB in 2020, and the utility of the indicators in setting concrete targets, a key World Bank eco- nomic expert interviewed observed “the impact gap between advances in [DB] rank and growth, employment and productivity. … Growth slowed. Job growth slowed.” Nor did investment clearly respond to improved indicators. A World Bank private sector development expert interviewed by IEG observed, “We don’t see a correlation between movement up in the [DB] ranking and the daily life of enterprises.” » In India (which moved from 133rd in DB2010 to 63rd in DB2020), business groups and foreign investors interviewed stated that they did not feel the benefits of strong movements in the DB indicators. One complaint Independent Evaluation Group World Bank Group    47 was that the scope of DB reforms did not address a number of binding constraints, ranging from input markets (land, labor) to bureaucracy to constrained competition. » For Rwanda, which moved from 67th in DB2010 to 38th in DB2020, the persistent structural constraints cited in the Country Program Evaluation (CPE) explained why, despite many measured reforms, gross domestic prod- uct per capita had not responded as intended and the Vision 2020 objec- tive of becoming a middle-income country was not achieved (World Bank 2019c). Persistent structural constraints narrowly concentrated the benefits of economic growth and major DB achievements. They included a “complex political economy environment … limiting fair competition and effective im- plementation of regulatory reforms … with companies closely affiliated with the government, the ruling party, and the military playing a dominant role in the private sector” (World Bank 2019c, 3). As a result, despite reforms, there was limited development of manufacturing, diversification of exports, and development of financial and services sectors. This disconnect was apparent in access to credit, where there were seven recorded DB reforms and where Rwanda rose to the rank of fourth in the world. A 2019 enterprise survey indicated that only 12 percent of firms reported having bank financing for in- vestment (14 percent for working capital), and access to finance was the most cited leading constraint. » In China, which moved from 89th in DB2010 to 31st in DB2020, analytic efforts were unable to capture the economic impact of remarkable progress The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Chapter 3 in DB-related reforms. Donors, foreign investors and experts interviewed indicated that the DB agenda did not capture some key business environment challenges, ranging from issues in the judiciary system to intellectual prop- erty rights and competition to difficulties with the financial system. World Bank concluded it was time to move the agenda forward to business envi- ronment challenges beyond the DB agenda. It also emphasized the need to strengthen access to data, and feedback from the private sector. » In the Democratic Republic of Congo, where EoDB rank was 182nd of 183 in 2010 and 183rd of 190 in 2020, lackluster overall performance was attributed to “negative reforms” for which some DB indicators showed a worsening of conditions, as well as to a communication deficit between the government and private sector on implementation of reforms, changes in the DB meth- odology during the case study period, and the failure of DB ratings to reward “complementary reforms.” Between DB2010 and DB2020, the Democratic Re- public of Congo was credited with 27 positive reforms and 8 negative reforms. » In Jordan, which moved from 100th to 75th in EoDB, the case study found it hard to identify direct economic benefits of the DB-informed reforms. Cited factors include Jordan’s susceptibility to regional shocks, macroeconomic policies, and the nature of economic growth, which, in turn, has not been in labor-intensive sectors. The study found that although DB-informed projects target areas that need to be reformed, reforms in some cases improved rank- ing without addressing the private sectors’ needs. 48 » Russia moved from 120th in EoDB in 2010 to 28th in EoDB in 2020, yet the case study found that movement of DB indicators was not informative on intermediate outcomes and impacts of reforms. Even though Russia moved from 182nd of 183 countries in DB2010 to 26th of 190 countries in DB2020 for DWCP, some indicators of constraints increased in business surveys. Bank staff pointed to a host of priorities outside even the expanded regulatory agenda growing out of DB, including SMEs’ ability to engage with global value chains, competition (antimonopoly) regulation, investment promotion, and more. Similarly, IEG’s deep dives found mixed effectiveness in specific indicator areas (table 3.1). Except for the paying taxes subindicator on tax rates, the reforms encouraged by DB-related activities were beneficial. At the same time, benefits could be limited when the reforms focused only on what was captured by the indicator and the base case scenario rather than addressing the broader regulatory area.  ey Findings on Effectiveness from Doing Business Indicator Table 3.1. K Deep Dives DB Indicator Summary Findings Starting a business » Country experiences in reforms are decidedly mixed given the many constraints to entry and disincentives to formal- ization. » There is general empirical support in the literature for Independent Evaluation Group World Bank Group    49 reducing the cost and complexity of business entry, but many countries experience only short-term or limited benefits from reforms. » For informal firms, there is evidence that the cost of regis- tration alone may not tip the decision to formalize, unless it is accompanied by additional incentives such as simpli- fied or reduced taxation or enhanced access to finance and land (World Bank 2013b). (continued) DB Indicator Summary Findings Getting credit » Useful in focusing reforms on movable collateral and credit information. » Deeper reforms take time. Where client countries focused on carrying out quick reforms to improve DB rankings, it overshadowed the long duration of time needed to set up adequate credit infrastructure. In Morocco, collateral law and registry reforms took nine years. It takes up to two years to set up a credit registry, and it is recommended to collect at least two years’ worth of data before launch. » The outcome of reforms depends heavily on whether there are other factors limiting credit, such as heavy presence of the state, weak competition, other distortions in credit markets, or demand-side factors constraining the flow of credit-worthy projects. The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Chapter 3 Trading across borders » The trade facilitation features captured by this indicator are linked to trade flows. » By focusing only on a single export and single import at the largest port in the largest business city in most coun- tries, the trading across borders indicator is less effective at capturing implementation progress in trade facilitation reform. » A lack of granularity means many reforms may not be picked up. For example, over the evaluation period, thorough trade reform in the Lao People’s Democratic Republic yielded only two recorded trading across bor- ders reforms in DB. Dealing with construction » In several countries, DWCP has proven to be a useful start- permits ing point for conversations about construction regulation. » Because important aspects (including corruption and informality) are missed, reforms are often not experienced as improvements by firms. » There is a risk that countries focus only on improving their DWCP ranking rather than overcoming persistent industry obstacles. » In China and the Russian Federation, it led to successful reform efforts. Those experiences indicate that consider- able supplemental analysis, expertise, and local under- standing at the municipal level must be mobilized to successfully introduce reforms. (continued) 50 DB Indicator Summary Findings Paying taxes » This indicator focused countries more on tax policy (rates) than on compliance costs, with 60 percent of tracked reforms in this area. » Of DB-tracked country reforms on tax rates, 46 percent were “negative”: tax payment burden increased. By con- trast, the tax administration component tracked reforms that were 95 percent positive. » The base case scenario can miss the actual regime under which many firms operate. Most small and medium enter- prises do not make ceramic flowerpots. » The hypothetical company in question does not qualify for any investment incentives or any benefits aside from those relating to its age or size. Yet in many countries, cor- porate tax codes abound in incentives, exceptions, and special provisions that can vary by sector, locality, and more. And the assumptions may miss important reforms: for example, when Colombia introduced a simplified re- gime for micro and small enterprises (SIMPLE) benefiting most small and medium enterprises, “paying taxes” did not count it as a positive reform. Source: Independent Evaluation Group; World Bank 2013b. Note: DB = Doing Business; DWCP = dealing with construction permits. IEG’s exploratory econometric analysis suggests that it is difficult to find significant, systematic relationships between changes in DB indicators and measurable outcomes such as gross domestic product growth, employment, Independent Evaluation Group World Bank Group    51 foreign direct investment, trade, or labor productivity. The econometric exercise tested the attribution of economic outcomes to movements of DB indicators. Many apparent correlations vanish when control variables are introduced. There is a high sensitivity to which variables were included (with an implicit sensitivity to omitted variables) and to small changes in mod- el specification. For example, although one model specification showed a significant relationship between the protection of minority investors in- dicator and foreign direct investment, another showed no significance. In some cases, DB indicators bore a counterintuitive but significant negative relationship with some outcome variables (for example, a negative relation- ship between resolving insolvency and employment, and between register- ing property and employment). Simple before-and-after analysis does not control for a host of explanatory factors, yet it can also be hard to specify a control group required to apply a more rigorous difference-in-difference technique. For these reasons, IEG is not offering its own econometric treat- ment of these relationships, deferring instead to the literature. World Bank Group Country Strategies An examination of a sample of IEG’s CPEs and reviews of Completion and Learning Report Reviews shows both the power and limits of the use of DB in country strategies.3 IEG’s review of a sample of IEG-evaluated country strategies indicates that, of 38 that proposed a DB-related work program, 74 percent of them achieved or mostly achieved the corresponding DB objectives. The highest success rate was achieved in low-income countries The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Chapter 3 (86 percent), followed by lower-middle-income countries (79 percent), and then upper-middle-income countries (64 percent). By region, although Latin America and the Caribbean, Sub-Saharan Africa, and Europe and Central Asia saw more than 75 percent of their relevant interventions achieve their objectives, this rate fell to 67 percent in South Asia, 50 percent in Middle East and North Africa, and 33 percent in East Asia and Pacific. However, achievement of project or country program objectives often does not translate into improvements in DB indicators. Although DB indicators are the most popular source used in country strategy documents to show progress on the business environment, they do not always prove respon- sive to the reforms being supported. Despite the 74 percent success rate of DB-informed country strategies in achieving their business environment re- form objectives, only 45 percent also showed improvements in DB indicators. A wide range of country experiences link DB-related reforms with improved business environments and sought-after economic benefits, but they also reveal mixed evidence about the effectiveness of DB-led reforms and the extent to which actual economic progress is achieved: » DB’s direct influence was evident in the Philippines. With IFC support, the country enacted a law known as the Ease of Doing Business and Efficient Government Service Delivery Act of 2018, which aimed to reduce processing time, cut bureaucratic red tape, and eliminate corrupt practices. Romania’s 52 Completion and Learning Report Review cites DB-inspired reforms as evi- dence of an improved business environment. » Some other countries achieved desired DB-related reforms but without evi- dent economic benefits. The Mexico CPE notes that DB-informed national and subnational initiatives seem linked to “a trend improvement” in the EoDB as well as to other reforms, including in competition policy (IEG 2018, xii). (None- theless, the country suffered from persistent low total factor productivity.) In Mauritius, DB reports and investment climate assessments were found to be “instrumental in highlighting areas of weakness and strength in the regulatory environment” and defining reform priorities. Mauritius improved its DB rank- ing yet “[d]espite this progress, there remain areas of weakness in the regula- tory environment” such that reforms “have not resulted in a surge in business registrations” (IEG 2016, 32–33). » Despite a focus on improving DB standing, some countries slipped back in DB ranking. In Bhutan, a 2017 performance and learning review found that “While Bhutan’s 2017 Doing Business ranking is the highest among South Asian countries, its drop in ranking from 71 to 73 (out of 190 countries) sug- gests that continued effort is needed to improve the business climate” (World Bank 2017c, 9). » In Zambia, the country achieved measured DB reforms with World Bank and IFC support, yet there was “a marginal decline in Zambia’s distance to frontier Independent Evaluation Group World Bank Group    53 score for overall ease of doing business” (World Bank 2019e, 6). For Benin, the DB target of reducing days to enforce a contract was detached from the country program in that “no program interventions could reasonably be identified with … number of days to enforce a contract—and hence any attribution is an issue” (World Bank 2018a, 6). » In other countries, the focus on DB had limited impact because it failed to address binding constraints. As described in chapter 2, this was the case in Rwanda, where the CPE found that, in spite of its top “DB reformer” status, “sustaining growth and poverty reduction—from already impressive achieve- ments—will require significant structural change in the economy” (World Bank 2019c, 3). In Albania, which reached the top half of the global ranking in Doing Business by DB2018, the CPE says it did not address important private sector constraints, including weak institutions and the absence of a reli- able and affordable power supply and adequate roads. In small states, IEG’s clustered program evaluation found that the Bank Group “needs a sharper focus on the most binding business constraints, using sector-specific lenses. Engagement based on the DB framework and similar cross-cutting approach- es led to useful reforms … but engagement did not always focus on the most binding constraints.” (World Bank 2016a, 47). In the Kyrgyz Republic, a key lesson was that: “overarching PSD [private sector development] reforms have greater impact than changes in specific doing business indicators” and selec- tive interventions (World Bank 2018c, 2). » In some countries, although the EoDB ranking showed progress, there was no clear evidence that the investment climate improved. The evaluation of The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Chapter 3 Tunisia’s country program through 2013 references business surveys and analytic work that “pointed to the heavy handed and pervasive influence of the state, and to the lack of serious reforms in the onshore sector” (World Bank 2014b, 48). A key problem was “privilege and unequal application of the rules of the game” that constrained competition. Weak governance was man- ifested in “discretion in the application of laws and regulations, inefficient procurement processes, rigged privatization, declassification of public land and assets and improper use of public banks.” These factors were “binding constraints on domestic private investments.” The project portfolio informed by DB is generally successful in achieving stated project objectives. Of 137 IEG-validated evaluated projects involving 291 interventions (components), 87 were World Bank lending projects and 50 were IFC AS.4 (World Bank advisory services and analytics projects have no validated evaluation framework.) At the intervention level, the World Bank had a success rate of 85 percent (134 out of 157). At the project level, this success rate was 70 percent. Within World Bank lending instruments, inter- ventions in development policy loans and investment project financing had a similar success rate (85 and 86 percent, respectively), while specific invest- ment loans were more successful, with 91 percent achieving their outcomes. Success rates were virtually identical in low-, lower-middle-, and upper-mid- dle-income countries. For IFC AS, the success rate for DB-informed inter- ventions was 78 percent, yet at the project level, it was 54 percent. Thus, for both, the overall success rate of DB-informed components was higher than 54 that of the projects that they were a part of. This suggests DB-informed com- ponent objectives may point to one appeal of DB-informed components— their high likelihood to succeed. In the evaluated portfolio, success rates were lower in 3 of 12 specific indi- cator areas requiring deep institutional reforms—enforcing contracts, regis- tering property, and resolving insolvency (figure 3.3). Projects on contracting with government and employing workers were extremely rare and even projects dealing with getting electricity and protecting minority investors were not common. Regarding how DB indicators were used, projects using the indicators as jus- tification were the most successful, followed by those using DB as a project indicator. Projects using DB as an objective or to generate indicators had somewhat lower rates of success (table 3.2). In terms of component objec- tives, setting up a reform agency was markedly less successful than others. Although components with other objectives were between 71 and 100 per- cent effective, reform agency components were only 57 percent effective for Bank lending, and 50 percent effective for IFC AS. Of the 291 evaluated interventions, 262 had data on immediate outcomes of the work, ranging from more transparent tariffs to improved systems to reduced costs of compliance. In general, the data showed better success in achieving project component objectives than in improving immediate outcomes. The measurements least likely to show the desired change were Independent Evaluation Group World Bank Group    55 improved administrative procedures (59 percent) and reduced days to com- plete procedures (63 percent). Fewer than half of the evaluated interventions provided data on intermedi- ate outcomes (figure 3.4). The most common of these to be reported were an increase in credit, cost savings for businesses, an increase in investment, and an increase in business entry. Of these, investment seemed to respond best to reforms (88 percent successful), followed by improved trade volume (67 percent) and business entry (64 percent). Other categories of intermedi- ate outcomes, including higher tax compliance or revenue (44 percent), cost savings for businesses (25 percent), and an increase in formality (17 per- cent), indicated success less often.  roject Success by Doing Business Indicator Area, Fiscal Years Figure 3.3. P 2010–20 Ease of doing business (n = 38) 89 Trading across borders (n = 39) 82 Starting a business (n = 60) 78 Getting credit (n = 44) 80 Dealing with construction permits (n = 33) 88 Paying taxes (n = 31) 87 Registering property (n = 19) 63 Enforcing contracts (n = 20) 65 The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Chapter 3 Resolving insolvency (n = 15) 53 Getting electricity (n = 6) 83 Protecting minority investors (n = 7) 100 Contracting with the government (n = 1) 100a Employing workers (n = 3) 100a 0 20 40 60 80 100 Share of successful interventions (%) Source: Independent Evaluation Group, portfolio review analysis. Note: Figure is based on 291 interventions (which excludes 12 interventions with no data regarding their intervention outcomes). Interventions may be counted more than once since they can support multiple business areas. The share of successful interventions is defined as the proportion of interventions that achieved or mostly achieved their intervention outcomes. a. Denotes n < 5 interventions.  uccess Rate of Doing Business–Related Interventions by Type Table 3.2. S of Use of the DB Report, Fiscal Years 2010–20 Use of DB Report or Indicators Indicators (no.) Success (%) As justification for project 183 82 As project indicator 142 80 As project objective 50 72 As support to generate DB report or 6 67 to update indicators Source: Independent Evaluation Group, portfolio review analysis. Note: Table is based on 291 interventions (which excludes 12 interventions with no data regarding their intervention outcomes). Interventions may be counted more than once since they can use DB reports in multiple ways. DB = Doing Business. 56  uccess Rate of Doing Business–Related Intermediate Figure 3.4. S Outcomes, Fiscal Years 2010–20 Increase lending (n = 28) 46 Cost savings for businesses (n = 20) 25 Increase investment (n = 16) 88 Increase business entry (n = 14) 64 Higher tax compliance or revenue (n = 9) 44 Increase formality (n = 6) 17 Increase trade volumes (n = 6) 67 Increase employment (n = 1) 100a Other (n = 50) 38 0 20 40 60 80 100 Share of successful interventions (%) Source: Independent Evaluation Group, portfolio review analysis. Note: Figure shows the percentage of interventions above the line, World Bank Group validated by the Independent Evaluation Group. Figure is based on 150 intervention with outcomes (which excludes 153 interventions with no data regarding their immediate outcomes). Interventions have multiple intermedi- Independent Evaluation Group World Bank Group    57 ate outcomes. The share of successful interventions is the proportion of interventions that achieved or mostly achieved their intermediate outcomes. a. Denotes n < 5 interventions. Learning from Factors of Project Success IEG’s review of evaluated projects found 696 references to factors to which project success or failure were attributed, most within the Bank Group’s control. Eighty-four percent of factors associated with project success and 82 percent of factors associated with project failure related to either quality at entry, project supervision, or monitoring and evaluation (figure 3.5): » Regarding quality at entry, the two most important factors were the role of accompanying or prior analytic work and proper identification of risks at appraisal. A negative factor was design complexity that exceeded implemen- tation capacity.5 » During supervision, key factors included client engagement and follow-up, effective coordination with internal and external stakeholders, and flexibility of implementation.6 » The quality of monitoring and evaluation could contribute to or inhibit success. 7 » Two external factors—client commitment and public sector institutional ca- pacity—figured most importantly, although agency coordination and political economy factors also mattered in select cases.8 » Advisory services and analytics self-evaluations indicate success is more The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Chapter 3 subject to external factors, led by political economy and agency coordination, and client commitment and capacity. Realism of the timetable, design sim- plicity, and elements of supervision ranging from client engagement to team composition were important internal factors.9 IEG’s econometric analysis finds several factors significantly predictive of project success (appendix H). Contextually, the multivariate logistic regres- sion found that, although country income level and region were not pre- dictors of success, the degree of political stability was. Key internal factors were good up-front analytic work, strong coordination, and appropriate team composition. IEG did not gain insight into DB country reform development outcomes applying similar econometric analysis. IEG’s deep dives into Project Performance Assessment Reports and IFC AS projects highlighted several lessons of success and failure, reinforcing the analysis above but adding nuance (table 3.3; appendix I). They add five key success factors: focus on binding constraints, a strong interagency coordi- nation unit, timely availability of expertise, longer-term and more compre- hensive engagements, and public-private dialogue. The analysis also shows how capacity could be built, through learning by doing, sustainable funding mechanisms, timely and appropriate expertise, and careful selection of con- tractors. Value is found in the use of complementary indicators and analyt- ics, as well as knowledge sharing and peer-to-peer learning. 58  actors Influencing Outcomes in IEG-Evaluated Projects, Figure 3.5. F Fiscal Years 2010–20 100 90 33 40 80 70 Percentage of IEG-evaluated projects 60 31 15 50 40 30 20 26 20 10 18 16 0 Adequate (n = 386) Inadequate (n = 310) Adequacy Independent Evaluation Group World Bank Group    59 Quality at entry Project supervision M&E considerations External factors Source: Independent Evaluation Group, analysis of evaluated projects and portfolio review analysis. Note: IEG = Independent Evaluation Group; M&E = monitoring and evaluation.  essons from World Bank Lending and International Finance Table 3.3. L Corporation Advisory Services Projects Success Failure 1. Strong ownership/commitment from 1. DB provides limited evidence on relevance, the government, coordinating ministry priority of reforms key (PPARs) needed to overcome inertia, vested interests. 2. Strength of interagency coordination 2. Mismatch between project complexity and unit is key. client capacity hinders success. 3. Capacity is built by learning by doing, 3. Failure is more likely where there is a lack sustainable funding mechanisms; and of focus on binding constraints. with timely and appropriate expertise and careful selection of contractors. The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Chapter 3 4. Having the right technical expertise at 4. Governments need a proper framework the right time and place matters. for inter-governmental cooperation across agencies and central/regional government. 5. World Bank and IFC can complement 5. One-stop shops and single windows need each other through collaboration. World authority over the functions they combine, Bank Group organizational changes at requiring process simplification and “back-of- times helped or hindered success. fice re-engineering,” not just a simplified interface. 6. Deeper reforms require comprehen- 6. Discontinuity of counterparts, regime sive and long-term engagement. Repeat change, and shifts in influence of champions interventions can be strategic or merely can disrupt progress. opportunistic. 7. Many IFC AS projects use other primary 7. Global indicator standardization under indicators due to limited DB relevance, DB may be out of alignment with industry timeliness. standards. 8. Emerging lessons point to value of 8. Lender preference for immovable collat- knowledge sharing, peer-to-peer learning. eral, distrust, and technical issues may limit uptake of collateral registries. 9. There is value in public–private dialogue. Source: Independent Evaluation Group. Note: AS = advisory services; DB = Doing Business; IFC = International Finance Corporation. 60 As indicated in figure 3.2, there were 24 additional claims of benefits to Doing Business 1  reform that did not specifically identify what those benefits were. 2  Of the 89 claims, 14 (16 percent) did not mention a source, and 8 mentioned prior World Bank publications (including DB reports), while the remaining 67 claims referenced specific papers. These 67 claims made 117 references to published literature; however, 39 of those references were mentioned in multiple DB reports, leaving only 77 unique sources. Excluding eight general claim references related to overall improvements in areas tracked by the ease of doing business index, the total number of papers referenced is 69. Of these 69 references, only 31 could be found in the bibliographic database maintained by the DB team from 100 leading journals. Of these, 24 could be mapped to a specific business area where a clear link of intervention to outcome might be established. Of these, 10 were identified by the Inde- pendent Evaluation Group in its desk review of literature as providing relevant evidence on outcomes for specific indicators, while seven of these 10 were validated by multiple articles (box 3.2). Two of the 12 suffered from mixed evidence where one finding contradicted another in the literature pool. Taking the structured literature review as a basis for rigorous study of outcomes, only 8 (13 percent) of the 69 references to published literature can be found in the structured literature review. 3  This section draws from the following Country Program Evaluations (CPEs) and Completion and Learning Report Reviews (CLRRs) and a Performance and Learning Review: Albania CPE (World Bank 2021a), Benin CLRR (World Bank 2018a), Bhutan Performance and Learning Review (World Bank 2017c), Independent Evaluation Group World Bank Group    61 Cluster CPE on Small States: Organisation of Eastern Caribbean States (World Bank 2016b), Kyrgyz Republic CLRR (World Bank 2018c), Cluster CPE on Small States: Mauritius (World Bank 2016a), Mexico CPE (World Bank 2018d), Philippines CPE (World Bank 2019d), Romania CLRR (World Bank 2018e), Rwanda CPE (World Bank 2019c), Tunisia CPE (World Bank 2014b), Zambia CLRR (World Bank 2019e) 4  Information was drawn from Implementation Completion and Results Report Reviews, Expanded Project Supervision Reports, and evaluative notes. Effectiveness, learning, and en- vironmental and social aspects sections apply only to projects evaluated by the Independent Evaluation Group. Twelve additional interventions were identified within these projects for which no relevant data were available on effectiveness. 5  Positive examples: Senegal Economic Governance Project (P113801; analytic work); Nigeria’s Lagos State Development Policy Operation II Project (P123352; identification and mitigation of risks). Negative examples include Côte d’Ivoire’s Second Poverty Reduction Support Credit Project (P143781) and Pakistan’s ADR Phase 2 (inadequate prior analysis—where the project suffered from insufficient up-front analysis of stakeholders as part of its due diligence, and Sier- ra Leone Financial Sector Support TA Project (P121514) which suffered from design complexity. 6  Positive examples: IFC Liberia Investment Climate Advisory Services Phase 3 Project (577647; engagement with counterparts); IFC’s Costa Rica Secured Transactions and Collat- eral Registries Project (coordination with a broad range of stakeholders); Tajikistan Private The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Chapter 3 Sector Competitiveness Project (P130091; adaptation to challenges). 7  Positive example: Mauritius Fourth Trade and Competitiveness Development Policy Loan (P116608; well-designed monitoring and evaluation [M&E] system). Negative examples: Kyrgyz Republic’s Development Policy Operation 1 Project (P126034; weak M&E coordination and integration); Togo’s Private Sector Development Support Project (P122326; weak M&E oversight and coordination). 8  Positive examples: Ukraine’s 2015 Second Development Policy Loan (P151479; client commit- ment); Colombia’s 2014 Taxes Program (599785; client commitment, stakeholder buy-in). Negative examples: Senegal Economic Governance Project (P113801; client commitment); Philippines’ 2014 Third Development Policy Loan (P147803; capacity constraints); IFC Investment Climate in the Caribbean Advisory Services Project (567627; limited counterpart skills, capacity). 9  Unlike World Bank lending and IFC advisory services, the framework for World Bank advi- sory services and analytics self-evaluation has neither been agreed to nor validated by the Independent Evaluation Group. Furthermore, the potential biases of unvalidated self-evalua- tion are evident from the overwhelming reported effectiveness rate of reimbursable advisory services (96 percent). 62 4 | Recommendations to Improve the Use of Doing Business in Country Reforms Highlights Doing Business (DB) indicators are most relevant to motivating countries to reform their legal and regulatory environment for busi- ness and pointing to areas for reform within their coverage area. They are best used in conjunction with complementary analysis and indicators that ensure limited development resources are focused on binding constraints. DB indicators are less relevant as project-level objectives or success metrics. Although DB has both influence and value, it is vital that the resource be as accurate and informative as possible and that it learn from evidence. Indicators suffer from inadequate feedback loops from research and field experience to their design and application. DB reports have made many claims for the benefits of measured country reforms that go beyond rigorous or replicated evidence. By favoring supportive evidence and by not establishing strong criteria for filtering evidence, the reports open the door for critics of their ob- jectivity and accuracy, posing a reputational risk to the World Bank Group and potentially misleading clients and stakeholders.   63 The evidence presented in this evaluation shows that DB indicators are powerful motivators of countries to reform their legal and regulatory environment for business. The high-profile DB report and indicators and the competition bred by country ranking are nearly uniformly recognized as features that motivate countries to engage in legal and regulatory reforms to enhance their business environments. Pragmatically, most experts judged this benefit to outweigh some of the methodological imperfections in the in- dicators and their indexation and ranking. In an institution that emphasizes the contribution of the private sector to development, being able to engage countries in enhancing conditions for doing business matters. The indicators have further shown value in pointing to problematic areas for reform within the areas they cover. The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Chapter 4 As a guide to binding constraints, DB is limited by its own coverage and methods. The stylization and economy of scope and data collection that make DB possible as an annual report on 190 countries’ regulatory con- ditions also necessitate careful application. This evaluation shows that in many country contexts, the following is true: (i) other factors—including policy, structural, and institutional ones—are essential for progress on the business environment; (ii) there is a broader or deeper reform agenda than that captured by the DB indicators; and (iii) the DB methodology and base case scenario may not necessarily reflect local conditions. Given limited country reform bandwidth and resources, it is vital to contex- tualize the strongly motivating messages of DB rankings with complementary sources of information to guide country reform priorities. It is in the interest of the client countries and the Bank Group to focus on the reforms yielding the greatest benefit. The integration of DB with other sources of data and an- alytic guidance seems to work well. The more sophisticated country users of DB (for example, Colombia) and the most sophisticated Bank Group analyses (for example, recent CPSDs) do this as a matter of course. More generally, it is vital for all users of DB to bring other sources of evidence and analytic tools to bear in determining reform priorities. This would also resonate with earlier IEG recommendations (see World Bank 2015b; World Bank 2019b) about con- sidering the policy objectives, such as protecting public health and safety, not just the compliance costs, of regulations. Such a broader set of sources has 64 generally not been reflected in DB reform memos, but it is evident in many country cases over the broader reform dialogue. The Bank Group needs to provide such context in its guidance, not only in textual caveats in DB re- ports, but at all stages of engagement. Recommendation 1. In line with much existing practice, the Bank Group should continue to use DB to motivate client engagement and to assist in reform focus within its menu of regulatory areas—but only where the priority and nature of reforms are confirmed by complementary analytics. The intend- ed outcome of this recommendation is that when countries engage with the Bank Group on business environment reform, they are consistently offered guidance based on a balance of appropriate evidence sources and frameworks. Although the available evidence on the benefits of improving conditions measured by DB is mostly positive for development outcomes, strong ev- idence is limited by indicator area and type of reform. It is important, therefore, to recognize the limited understanding of the extent to which improvements in DB indicators result in improved development outcomes. Further, the evidence gathered in this evaluation shows that substantive re- forms can fail to move the indicators and that, even if they do, such reforms may not yield substantial development benefits. The same features that enable DB to be produced annually and globally limit its value as an indica- tor of progress and especially as an explicit objective of reform. By covering only one to two cities and an often-hypothetical specific company or trans- action, DB is made possible but also constrained. A common complaint from Independent Evaluation Group World Bank Group    65 client countries is that DB indicators are not granular enough to track their reform implementation. Given this lack of granularity, IEG found that IFC AS has largely stopped using DB indicators as primary monitoring indicators. In addition, establishing DB indicators as project objectives invites the kind of strategic behavior that narrows focus to just influencing the indicator, rather than addressing the full substantive area of reform. It can increase the kind of pressure on the Bank Group from clients to improve indicators that was identified in the Group Internal Audit Vice Presidency audit (World Bank Group 2020). Yet 42 percent of the identified portfolio used DB indicators as project objectives, project indicators, or both. Recommendation 2. Consistent with good practice, the Bank Group should avoid using DB indicators as explicit reform objectives or monitoring indi- cators in projects and country strategies and, where avoidable, should not use DB as primary indicators of reform progress. The intended outcome is to reduce a practice that ties project success to movement in DB indicators, which has proven difficult and potentially misleading, eliminating the usage of DB as a sole indicator or objective and focusing instead on more tailored and granular indicators of success. DB indicators currently suffer from inadequate feedback loops from research and field experience to their design and application. While it is important to recognize the influence and value of DB, it is also vital that the resource be made as accurate and informative as possible and that it learn from evi- dence. Complementary work by DEC’s expert committee will provide great- The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Chapter 4 er insights into potential improvements in the methodology of individual indicators. IEG found limitations in the extent to which indicators reflect evidence and experience. First, there are indicators for which there is little evidence regarding the economic relevance of what the indicators measure. DWCP and getting electricity are important areas for indicators, but there is no rigorous empirical support for outcomes linked to the aspects that DB indicators currently capture. Little rigorous evidence is available overall on getting electricity, registering property, and enforcing contracts. Three scholarly articles in the DB literature database question the ability of DWCP to capture the real difficulty of building regulations for domestic SMEs. This raises the question, why would the Bank Group continue to produce indi- cators and promote reforms based on them without firm evidence of their development benefits? Standard assumptions about how compliance costs and time are influ- enced by “steps” and “documents” may be challenged by digitalization and e-government. It is vital, therefore, that indicators keep up with changes in technology. For example, use of big data and the growth of digital financial services may erode the relevance of traditional credit information systems. Although this evaluation recognizes that changes to DB indicators have costs to client countries and to researchers, well-communicated and infre- quent changes to indicators can improve their accuracy without imperiling their benefits. This is confirmed by the generally benign view many experts have of several past reforms to indicators. In spite of the acknowledged 66 costs of discontinuities in indicators, underscored by the 2018 external audit (Morck and Shou 2018), there are ways to limit these costs. Given its influence, it is desirable for the DB approach to capture a fuller range of regulatory, legal, and institutional conditions that influence the life cycle of enterprises. IEG has examined this issue before (World Bank 2015b), as have many others, highlighting areas ranging from consumer and environmental protection to competition and intellectual property regulation. Even in areas nominally covered, aspects of some key regulations—like sectoral business licensing and many areas of contract law—remain out of scope. Recommendation 3. The Bank Group should update DB indicator areas and definitions at regular and predictable intervals to reflect learning from re- search and field experience. Doing so will improve links to important devel- opment outcomes, strengthen relevance to the experience of domestic SMEs, and adapt to technological changes in the areas covered by the indicators. The intended outcome is for the Bank Group to deliver the best possible infor- mation to country clients to inform their business environment reforms as guided by research and field experience. The indicator agenda and character- istics should reflect ongoing learning from evidence and experience, showing what matters and the need to adapt to technological changes in areas covered. Similarly, advice to client countries on appropriate reform models conveyed through the DB report, projects, and knowledge sharing should reflect such learning. Where review of evidence (like that undertaken for this evaluation) Independent Evaluation Group World Bank Group    67 reveals knowledge gaps, this has direct implications for a strategic program of research to fill such knowledge gaps. Effective feedback loops require both a steady flow of information from which to learn—from research and moni- toring and evaluation—and routines through which DB indicators can adapt to such feedback with the least disruption to users. Disruption caused by such updates can be managed by making such changes infrequently and predict- ably, by engaging in a transparent and consultative process, and by maintain- ing former indicator series for a period of years after changes are introduced. DB reports have made many claims for the benefits of measured reforms that go beyond rigorous or replicated evidence. By favoring supportive evidence and by not establishing strong criteria for filtering evidence, the reports open the door for critics to question the objectivity and accuracy of this important resource. This also poses a reputational risk to the Bank Group and Development Economics Vice Presidency and may mislead clients and stakeholders. This evaluation shows how DB reports, alongside some robust claims, forward some claims on the relationship of changes in areas mea- sured by its indicators to development outcomes that have no clear evidence, some with inadequate evidence, and some where evidence is mixed. The criteria for selecting and featuring such claims is not transparent or consis- tent, but the general tone is one of advocacy. This does not seem needed or advantageous given the traction DB has found worldwide and the consider- able evidence in the literature cited in this evaluation validating important parts of its agenda. IEG’s SLR shows that rigorous methods and evidence are available and implementable. Where there are gaps, it establishes an excel- The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Chapter 4 lent agenda for future research. The Bank Group need not set its criteria for selecting evidence at the level of the SLR, but it should establish, publicize, and apply its criteria. Recommendation 4. The Bank Group should strengthen the accuracy and validity of DB claims in DB reports and related communications in line with robust evidence. The intended outcome is to produce DB reports that accu- rately inform their audience about the relationship of DB indicators to out- comes based on robust evidence. As a leading publication from the research arm of the World Bank, the Bank Group needs to ensure that claims made in DB reports are robustly substantiated by research and refrain from claims that are not. It should adopt clear standards of evidence and commission DEC or outside research if evidence is lacking. The ultimate outcome sought with this set of recommendations is to build on the many good practices observed in the course of this evaluation. In doing so, the Bank Group could ensure that the DB indicators maintain their substantial power of motivating and engaging client countries in business environment reform, in a manner that guides clients to prioritize the reforms with the greatest development benefits for their socioeconomic situation, based on a balanced and accurate consideration of evidence. 68 Bibliography Besley, Timothy. 2015. “Law, Regulation, and the Business Climate: The Nature and Influence of the World Bank Doing Business Project.” Journal of Economic Per- spectives 29 (3): 99–120. doi:10.1257/jep.29.3.99. Doing Business Report Independent Review Panel. 2013. Independent Panel Review of the Doing Business Report. Washington, DC: World Bank. http://pubdocs. worldbank.org/en/237121516384849082/doing-business-review-panel-re- port-June-2013.pdf. Doshi, Rush, Judith G. Kelley, and Beth A. Simmons. 2019. “The Power of Ranking: The Ease of Doing Business Indicator and Global Regulatory Behavior.” Interna- tional Organization 73 (3): 611–43. doi:10.1017/S0020818319000158. Haidar, Jamal Ibrahim. 2012. “The Impact of Business Regulatory Reforms on Economic Growth.” Journal of the Japanese and International Economies 26 (3): 285–307. doi:10.1016/j.jjie.2012.05.004. Kelley, Judith G., and Beth A. Simmons, eds. 2020. The Power of Global Performance Indicators. Cambridge: Cambridge University Press. Morck, Randall, and James Chenxing Shou. 2018. On the Integrity of the “Ease of Doing Business” Indicators: Final Report. Washington, DC: World Bank. https://www. Independent Evaluation Group World Bank Group    69 doingbusiness.org/content/dam/doingBusiness/media/Miscellaneous/Audit/Do- ing-Business-External-Audit-2018_Full-Report.pdf. OECD (Organisation for Economic Co-operation and Development). 2021. Applying Eval- uation Criteria Thoughtfully. Paris: OECD Publishing. doi:10.1787/543e84ed-en. Rwanda, Republic of. 2007. Economic Development and Poverty Reduction Strategy, 2008–2012 Shaping our Development. Kigali. https://www.imf.org/external/pubs/ ft/scr/2013/cr13360.pdf. Rwanda, Republic of. 2013. Economic Development and Poverty Reduction Strategy, 2013–2018 Shaping Our Development. Kigali. https://www.imf.org/external/pubs/ ft/scr/2013/cr13360.pdf. Stern, Nicholas. 2002. A Strategy for Development. Washington, DC: World Bank. https://openknowledge.worldbank.org/handle/10986/15213. World Bank. n.d. “Ease of Doing Business Scores.” World Bank. https://www.doing- business.org/en/data/doing-business-score. World Bank. 2003a. Doing Business in 2004: Understanding Regulation. Washington, DC: World Bank. https://www.doingbusiness.org/en/reports/global-reports/do- ing-business-2004. World Bank. 2003b. World Bank Group Private Sector Development Strategy: Imple- mentation Progress Report. Washington, DC: World Bank. http://documents. The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Bibliography worldbank.org/curated/en/322041468164040279/World-Bank-Group-pri- vate-sector-development-strategy-implementation-progress-report. World Bank. 2008. Doing Business: An Independent Evaluation: Taking the Measure of the World Bank–IFC Doing Business Indicators. Independent Evaluation Group. Washington, DC: World Bank. https://openknowledge.worldbank.org/han- dle/10986/6467. World Bank. 2009. Doing Business 2010: Reforming through Difficult Times. Washing- ton, DC: World Bank. https://www.doingbusiness.org/en/reports/global-reports/ doing-business-2010. World Bank. 2010. Doing Business 2011: Making a Difference for Entrepreneurs. Wash- ington, DC: World Bank. https://www.doingbusiness.org/en/reports/global-re- ports/doing-business-2011. World Bank. 2011. Doing Business 2012: Doing Business in a More Transparent World. Washington, DC: World Bank. https://www.doingbusiness.org/en/reports/glob- al-reports/doing-business-2012. World Bank. 2012. Doing Business 2013: Smarter Regulations for Small and Medi- um-Size Enterprises. Washington, DC: World Bank. https://www.doingbusiness. org/en/reports/global-reports/doing-business-2013. World Bank. 2013a. Doing Business 2014: Understanding Regulations for Small and Medium-Size Enterprises. Washington, DC: World Bank. https://openknowledge. worldbank.org/handle/10986/16204. 70 World Bank. 2013b. “Rethinking Informality: What is Keeping Firms from Go- ing Formal?” World Bank News. https://www.worldbank.org/en/news/ feature/2013/07/11/Rethinking-Informality-What-is-Keeping-Firms-from-Go- ing-Formal. World Bank. 2014a. Doing Business 2015: Going Beyond Efficiency. Washington, DC: World Bank. https://www.doingbusiness.org/en/reports/global-reports/do- ing-business-2015. World Bank. 2014b. Tunisia Country Program Evaluation, FY05–13. Independent Eval- uation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/ evaluations/tunisia-country-program-evaluation-fy05-13. World Bank. 2015a. Doing Business 2016: Measuring Regulatory Quality and Efficiency. Washington, DC: World Bank. https://www.doingbusiness.org/en/reports/glob- al-reports/doing-business-2016. World Bank. 2015b. Investment Climate Reforms: An Independent Evaluation of World Bank Group Support to Reforms of Business Regulations. Independent Evaluation Group. Washington, DC: World Bank. https://openknowledge.worldbank.org/ handle/10986/22724. World Bank. 2016a. Cluster Country Program Evaluation on Small States: Mauritius Country Case Study (FY07–15): Enhancing Competitiveness and Private Sector Development. Working Paper. Independent Evaluation Group. Washington, DC: Independent Evaluation Group World Bank Group    71 World Bank. https://openknowledge.worldbank.org/handle/10986/25195. World Bank. 2016b. Cluster Country Program Evaluation on Small States: Regional Program Evaluation of the Organisation of Eastern Caribbean States. Independent Evaluation Group. Washington, DC: World Bank. https://openknowledge.world- bank.org/handle/10986/33199. World Bank. 2016c. Doing Business 2017: Equal Opportunity for All. Washington, DC: World Bank. https://www.doingbusiness.org/en/reports/global-reports/do- ing-business-2017. World Bank. 2016d. Supporting Transformational Change for Poverty Reduction and Shared Prosperity: Lessons from World Bank Group Experience. Independent Eval- uation Group. Washington, DC: World Bank. https://openknowledge.worldbank. org/handle/10986/24024. World Bank. 2017a. Data for Development: An Evaluation of World Bank Support for Data and Statistical Capacity. Independent Evaluation Group. Washington, DC: World Bank. http://documents.worldbank.org/curated/en/705581506620506441/ Data-for-development-an-evaluation-of-World-Bank-support-for-data-and-sta- tistical-capacity. World Bank. 2017b. Doing Business 2018: Reforming to Create Jobs. Washington, DC: World Bank. https://www.doingbusiness.org/en/reports/global-reports/do- ing-business-2018. World Bank. 2017c. “Performance and Learning Review of the Country Partner- ship Strategy for the Kingdom of Bhutan for the Period FY2015–2019.” Report The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Bibliography 110736. Washington, DC: World Bank Group. http://documents.worldbank.org/ curated/en/551031496887225781/Bhutan-Performance-and-learning-review- of-the-country-partnership-strategy-for-the-period-FY2015-2019. World Bank. 2018a. “Benin—Review of Completion and Learning Review for FY13– 18.” Completion and Learning Report Review, Independent Evaluation Group, World Bank, Washington, DC. https://ieg.worldbankgroup.org/sites/default/files/ Data/reports/clrr_beninfy13-18.pdf. World Bank. 2018b. Doing Business 2019: Training for Reform. Washington, DC:  World Bank. https://www.doingbusiness.org/en/reports/global-reports/do- ing-business-2019. World Bank. 2018c. “Kyrgyz Republic—Review of Completion and Learning Review for FY14–17.” Completion and Learning Report Review, Independent Evaluation Group, World Bank, Washington, DC. https://ieg.worldbankgroup.org/sites/de- fault/files/Data/reports/clrr_kyrgyz1417.pdf. World Bank. 2018d. Mexico: Country Program Evaluation: An Evaluation of the World Bank Group’s Support to Mexico (2008–17). Independent Evaluation Group. Washington, DC: World Bank. doi:10.1596/30937. World Bank. 2018e. “Romania—Review of Completion and Learning Review for FY14–18.” Completion and Learning Report Review, Independent Evaluation Group, World Bank, Washington, DC. https://ieg.worldbankgroup.org/sites/de- fault/files/Data/reports/clrr_romaniafy1418.pdf. 72 World Bank. 2018f. Data for Development: An Evaluation of World Bank Support for Data and Statistical Capacity. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/sites/default/files/Data/Evaluation/ files/datafordevelopment_0.pdf. World Bank. 2019a. Doing Business 2020. Washington, DC: World Bank. https://www. doingbusiness.org/en/reports/global-reports/doing-business-2020. World Bank. 2019b. Grow with the Flow: An Independent Evaluation of the World Bank Group’s Support to Facilitating Trade 2006–17. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/sites/default/ files/Data/Evaluation/files/facilitatingtrade.pdf. World Bank. 2019c. Rwanda Country Program Evaluation, FY09–17. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup. org/evaluations/rwanda-country-program. World Bank. 2019d. The World Bank Group Partnership with the Philippines, 2009–18: Country Program Evaluation. Independent Evaluation Group. Washington, DC: World Bank. doi:10.1596/33204. World Bank. 2019e. “Zambia—Review of Completion and Learning Review for FY13–17.” Completion and Learning Report Review, Independent Evaluation Group, World Bank, Washington, DC. https://ieg.worldbankgroup.org/sites/de- fault/files/Data/reports/clrr_zambiafy13-17.pdf. Independent Evaluation Group World Bank Group    73 World Bank. 2021a. Albania Country Program Evaluation. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/sites/de- fault/files/Data/Evaluation/files/Albania_CPE.pdf. World Bank. 2021b. Doing Business Indicators and Country Reforms. Approach Pa- per. Independent Evaluation Group. Washington, DC:  World Bank. https://ieg. worldbankgroup.org/sites/default/files/Data/reports/ap_doingbusiness.pdf. World Bank. Forthcoming. Doing Business and Country Reforms. Issues Paper. Inde- pendent Evaluation Group. Washington, DC: World Bank. World Bank Group. 2020. “Data Integrity in the Production Process of the Doing Business Report.” Assurance Review Report FY21-2-2-2103656. Group Internal Audit, World Bank, Washington, DC. http://documents.worldbank.org/curated/ en/134831608154762985/pdf/Data-Integrity-in-Production-Process-of-the-Do- ing-Business-Report-Assurance-Review.pdf. The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Bibliography 74 APPENDIXES Independent Evaluation Group The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20 Appendix A. Methodology Evaluation Objective The evaluation explored the accuracy and usefulness of the Doing Business (DB) report and indicators in identifying reform priorities, guiding the de- sign of reforms, and monitoring the progress of reforms. It probed available sources for evidence of the benefits of DB-related reforms for improving investment and employment, inequality, and other country development objectives. It also gathered and reported available information on the social The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix A and environmental consequences of DB-related reforms, the comprehen- siveness of DB as a guide to investment climate reform, and the value (and challenges) of aggregated DB indicators, including the “ease of doing busi- ness” score. The evaluation objective inspired two lines of inquiry that guided data col- lection and analysis (box A.1). Evaluation questions were designed to break the topic into tractable components in the areas of DB influence, relevance, and effectiveness, and factors affecting the outcomes of DB support. Box A.1. The Evaluation’s Two Lines of Inquiry Doing the Right Things: This line of inquiry examined the relevance of indicators to country contexts and priorities, substantive dimensions of the areas they cover, and World Bank Group strategic and operational priorities. Doing Things Right: This line of inquiry examined whether Doing Business has been used effectively by the Bank Group and client countries to achieve intended imme- diate and intermediate business environment reform outcomes, subject to broader policy priorities. Source: Independent Evaluation Group. 78 Evaluation Questions The evaluation sought to answer the questions in table A.1 under an overar- ching question about lessons from the use of DB in guiding business envi- ronment reforms in client countries over the period of fiscal years 2010–20 (FY10–20). The evaluation questions were informed by a review of literature, previous Independent Evaluation Group (IEG) evaluations, and consulta- tions with subject matter experts in business environment reform. Table A.1. Evaluation Questions Data Collection and Strengths and Limita- Evaluation Questions Analysis Methods tions Is DB doing the right things in terms of specific indicators and country contexts? 1. Doing the Right Things: Literature review; synthe- » Volume and breadth of What is the relevance of DB sis of available evaluative available literature. indicators to materials and country » Access to new tools to » The business envi- case studies; review of enhance identification of ronment priorities of country strategies, diag- correct portfolio. client countries? nostics, and evaluations; » Limited external, inde- » The substantive di- PRA of approved projects; pendent evaluations of mensions within each prior IEG country-based DB use, with many gaps. indicator’s focus area? evaluations; deep dives for individual DB areas; » Inconsistent sources on » World Bank Group use of DB indicators in statistical and economet- strategic and opera- client country reforms, ric analysis. tional priorities? Independent Evaluation Group World Bank Group    79 with many gaps. » COVID-19 limitations on fieldwork. (continued) Data Collection and Strengths and Limita- Evaluation Questions Analysis Methods tions Is DB being used by clients and the Bank Group in the right ways to achieve the best effect for business environment reform? 2. Doing Things Right: Is DB Structured literature » Consistent triangulation » Effectively achieving review, PRA of evaluated between quantitative desired outcomes? projects, including PPARs; and qualitative method- » Being used by clients review of country-based ologies to draw findings and the Bank Group evaluations; country case and conclusions. to achieve the best studies; expert and stake- » Country case studies outcomes for business holder interviews; analysis, selected based partly on environment reform (for deep dives for individual the size and diversity of example, entry, invest- DB areas; statistical and interventions supported ment, and employment) econometric analysis of or not supported by the The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix A subject to broader policy DB and external indicators Bank Group. priorities (for example, of effectiveness. » Biases inherent in inter- public safety, equity, and views. the environment)? » Project evaluations rarely showing outcomes or whether reforms were sustained. » COVID-19 limitations on fieldwork. Source: Independent Evaluation Group. Note: DB = Doing Business; IEG = Independent Evaluation Group; PPAR = Project Performance Assess- ment Report; PRA = portfolio review and analysis. Overarching Principles and Methods Design Three central principles motivated the evaluation design: theory-based evaluation, mixed-methods analysis, and multilevel analysis. First, the evaluation adopted a theory-based approach that sought to understand the linkages between interventions supported by DB, World Bank Group, or client governments, and country-level reforms based on the influence model elaborated. Second, the evaluation also applied a mixed-methods approach that combined an array of complementary methods for data collection and analysis (for example, internal project-level data, external country data sets, project performance data, semistructured interviews, case studies, sector deep dives, and literature reviews), and then triangulated to ensure robust findings. Third, the evaluation used three levels of analysis—global, country, 80 and project or intervention level (figure A.1), which provided information on whether and how country reforms and Bank Group programs and projects that were related to DB succeeded or not, including factors associated with their success or failure. Figure A.1. Products, Methods, and Levels of Analysis Source: Independent Evaluation Group. Note: AP = Approach Paper; CLRR = Completion and Learning Report Review; CPE = Country Program Independent Evaluation Group World Bank Group    81 Evaluation; DB = Doing Business; IFC-AS = International Finance Corporation advisory services; PPAR = Project Performance Assessment Report. The global level of analysis was supported by the structured literature re- view, analysis of Bank Group strategies and policies, deep dives into a subset of areas measured by DB, an expert panel discussion, review of DB report affirmative claims, review of prior evaluations, structured interviews of key informants, and a quantitative analysis of the DB indicators and indexes. At the country level, the evaluation collected data and evidence through the following methods: Bank Group country strategy and diagnostics analy- sis, country case studies, comparative case analysis, and review of relevant Country Program Evaluations and Completion and Learning Report Reviews (CLRRs; figure A.1). At the project or intervention level, a portfolio review analysis and deeper reviews of International Finance Corporation advisory services (IFC AS) evaluated projects and of evaluated Project Performance Assessment Reports for World Bank lending projects were conducted to understand the breadth of interventions undertaken by the Bank Group to support DB reforms. The evaluation produced two deliverables: » First, an Issues Paper that took stock of existing evidence on DB relevance, identified key issues, and elaborated a framework and testable hypotheses for the deeper analysis of a full evaluation to follow. » Second, this focused evaluation that applies the theory of change and lines of inquiry (and their explicit hypotheses) generated in the first product (ta- The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix A ble A.2). This product answers questions about relevance and effectiveness in terms of both immediate and intermediate outcomes. Table A.2. Evaluation Questions, Lines of Inquiry, and Hypotheses Evaluation Question Line of Inquiry Explicit Hypotheses 1. What is the rele- Whether all DB indicators are DB indicators vary in their rel- vance of DB indicators relevant to constraints influ- evance to the key constraints to encing business dynamics influencing business dynamics, a. The business envi- and economic contribution of hence the economic contribu- ronment priorities of policy reforms. tion of reforms, based on the client countries? area and the country context. b. The substantive dimensions within each indicator’s focus area? c. World Bank Group strategic and opera- tional priorities? 1a and 2b Whether (i) client institutional Client institutional capacity Are clients using DB capacity and ability (high vs. and capability is a key factor to achieve improved low) to tailor the use of DB influencing how DB indicators outcomes for business indicators to its development should be used and whether environment reform in priorities and framework their use is effective. line with the country’s and (ii) client motivation (for Client motivation is a key factor development policy example, to achieve policy influencing the how DB indica- priorities? reform vs. improve DB rank- tors should be used and wheth- ing) are key influences on the er their use is effective. use and appropriateness of DB indicators. (continued) 82 Evaluation Question Line of Inquiry Explicit Hypotheses 1a, 1b, 2b Whether DB indicators are DB indicators are useful to draw useful to (i) draw attention attention and resources to legal and resources to business and regulatory reforms in the environment reform in areas areas they directly measure. they measure directly, (ii) DB indicators are useful to draw attention and resourc- motivate broader processes of es to business environment business environment reform. reform beyond the areas they DB indicators are useful for measure directly, (iii) correctly identifying binding constraints identify and order priority and priority reforms. reform areas, and (iv) monitor DB indicators are useful to and evaluate reform imple- monitor and evaluate reform mentation. progress. 1a, 1b, 2a, 2b Whether complementary DB indicators are more effec- data and guidance enhance tively used in conjunction with the relevance of DB indica- complementary data and guid- tors to business environment ance to enhance the relevance, reforms and the ability to comprehensiveness, and depth follow with complementary of corresponding business envi- or deepening reforms. ronment reforms. 2a. Is DB reform Whether the feedback loops DB benefits from robust feed- achieving desired between available empirical back loops between empirical outcomes? evidence and DB indicator evidence (research and evalua- And derives from design and use are robust. tive data) and indicator design. feedback loop in theo- ry of change. 1c. and derives from Whether current institutional Current institutional arrange- challenges regarding arrangements separating ments separating DB indicator data integrity. indicator generation from generation from DB reform proj- Independent Evaluation Group World Bank Group    83 project work (including reim- ect work (including reimburs- bursable assistance) are ade- able assistance) are adequate to quate to prevent a perception prevent a perception of conflict of conflict of interest within of interest within the World Bank the World Bank Group. Group. Source: Independent Evaluation Group. Note: DB = Doing Business. Evaluation Components As noted, this IEG evaluation uses several evaluation tools and methods to understand and assess the relevance and effectiveness of the use of DB indicators at the global, country, and project or intervention levels. In prepa- ration, each line of inquiry identified in the Issues Paper and its subordinate evaluation hypothesis was explored applying a constellation of tools to pro- vide evidence. Table A.3 maps the tools that were most helpful in illuminat- ing the lines of inquiry. The Approach Paper has highlighted the limitations of each of these methodologies, and the need and intention for triangulation to enhance the robustness of findings. Potential error is limited by drawing evidence from multiple sources to test each hypothesis. Table A.3. Lines of Inquiry and Evaluation Tools Evaluation Tools Used to Test Hypotheses The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix A Main Lines of Inquiry PRA SLR IDD CCS CLR SSI SER 1. All DB indicators are relevant/are not relevant to constraints influenc- X X X X X X ing business dynamics and econom- ic contribution of policy reforms. 2. The following factors are/are not key at influencing the use and appropriateness of DB indicators include a. Client institutional capacity and ability (high vs. low) to tailor use of X X X X X indicators to its development priori- ties and framework; b. Client motivation (for example, achieve policy reform vs. improve X X X X X DB ranking). 3. DB indicators are useful/not useful for a. Drawing attention and resources to business environment reform in X X X X areas they measure directly; b. Drawing attention and resources to business environment reform X X X X beyond the areas they measure directly; c. Correctly identifying and ordering X X X X priority reform areas; d. Monitoring and evaluating reform X X X X X implementation. 4. Complementary data and guid- ance enhance/do not enhance the relevance of DB indicators to busi- X X X X ness environment reforms and the ability to follow with complementary or deepening reforms. 84 (continued) Evaluation Tools Used to Test Hypotheses Main Lines of Inquiry PRA SLR IDD CCS CLR SSI SER 5. The feedback loops between em- pirical evidence and indicator design X X X X and use are robust/weak. 6. Current institutional arrangements X X separating indicator generation from project work are adequate/inad- equate to prevent a perception of conflict of interest. Source: Independent Evaluation Group. Note: DB = Doing Business; CCS = country case studies; CLR = country-level reviews of strategies and diagnostics; IDD =  indicator-specific deep dives; PRA = portfolio review and analysis; SER = statistical and econometric analysis; SLR = structured literature review; SSI = semistructured interviews. Portfolio Review and Analysis The evaluation conducted a systematic desk review and assessment of projects to identify design features and characteristics, achievement of objectives, drivers of success and failure, and achievement of social objec- tives. Consistent with the Approach Paper, a DB-informed project was one that referenced DB indicators in its justification, objectives, or indicators. A combination of manual identification, manual codification, and supervised machine learning methods was used to identify all Bank Group projects Independent Evaluation Group World Bank Group    85 approved between FY10 and FY20 that were informed by the DB report or indicators in their objectives, components, and result matrices, and that had relevant influence in their motivation (figure A.2). Feedback from the World Bank and IFC was also incorporated in the final portfolio identification. The overall objective of the identification methodology was to classify projects that referenced DB indicators or relevant dimensions in their ratio- nale, objectives, components, or result matrices. The manual identification process is the method traditionally used by IEG; however, the identification of projects was challenging because the team could only extract information about projects’ objectives and titles, and project result matrices from the data warehouse. In this sense, some projects qualified as being informed by DB were still missing. Thus, IEG decided to also undertake a pilot automa- tized process using supervised learning methods, and experiment with what this second identification process would indicate. IEG manually reviewed the sample to assure the identified projects qualified. A comparison between both categorizations allowed IEG to assure full project-level description while assessing the pilot’s accuracy and pertinent improvements. The proto- col guiding this process is outlined in the following sections. Figure A.2. Identification Methodologies in Place The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix A Source: Independent Evaluation Group. Note: ASA = advisory services and analytics; DB = Doing Business; IFC = International Finance Corporation. Manual Identification First, IEG conducted a systematic keyword search of projects’ objectives and titles and looked for DB indicators and subindicators in projects’ result matrices. IEG selected those that contained at least one of the targeted key- words described in table A.4. 86 Table A.4. Keyword Search Strategy to Identify Doing Business Portfolio Area of Business Regulation Keywords Used General doing business/business environment/business regula- tion/business operation/regulatory reform Starting a business procedure/time/cost with start/starting Dealing with construction procedure/time/cost with construction permit(s) permits Getting electricity procedure/time/cost with electricity Registering property (property with procedure/time/cost) and (land with ad- ministration) Getting credit (bureau/registry/information with credit) and (movable collateral with laws) Protecting minority investors minority with rights Paying taxes paying taxes/tax payment/time to comply Trading across borders trading across borders and (time/cost/documents with import/export) Enforcing contracts time/cost with dispute Resolving insolvency time/cost/recovery rate/legal framework with insolvency Employing workers employment with regulation Contracting with the gov- (time/procedure with procurement) and (procurement ernment with regulatory framework) Source: Independent Evaluation Group, portfolio review analysis. Note: Of the projects identified through the keywords search strategy, 20 percent were identified based Independent Evaluation Group World Bank Group    87 on keywords included in the general category. When the project objective was not available in the business warehouse proj- ect list, IEG adopted an alternative strategy, according to the institution. For World Bank projects, if the project objective was not available in the business warehouse project list, IEG searched relevant themes for alignment with DB areas (see table A.5). World Bank advisory services and analytics (ASA) included the economic and sector work and technical assistance product lines from operations approved between FY10 and FY18, as these are the product lines that identify sectors and thematic codes. But because the five ASA product lines (economic and sector work, technical assistance, IW, TE, and PA) were replaced with the AA code after FY18, IEG used this code for the operations approved in FY19 and FY20. For IFC projects, if the project objective was not available, IEG used relevant business lines summarized in table A.5. Table A.5. Bank Group System Codes to Identify Doing Business Portfolio World Bank Lending and ASA International Finance Corporation Theme codes: Business line products and sector names: » Business enabling environment (21) » Economic and private sector development » Energy and policy reform (862) – climate business area » Access to energy (863) » A2F – credit bureaus (D) » Personal and property rights (423) » FAM – collateral registries/secured trans- actions (ID) » Financial infrastructure and access (32) The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix A » EFI – credit infrastructures (ENT) » Tax policy (114) » IC – investment policy (I-D) » Trade (14) » IC – discontinued product – access to land » Legal institutions for a market econo- (EXT) my (422) » IC – discontinued product – business » Judicial and other dispute resolution taxation (I-D) mechanism (421) » IC – trade logistics (I-D) » Labor markets institutions (662) » IC – debt resolution and business exit » Transparency, accountability and good (ENT) governance, procurement (432) » Investment climate – business regulation (D) » SBA – sustainable and inclusive investing (ENT) » SBA – environmental, social and trade standards (ENT) » TAC – business environment (D) » Credit information services (O-KB) Source: Independent Evaluation Group. Note: ASA = advisory services and analytics. This process identified 1,669 potential DB projects: 372 World Bank lending projects, 916 World Bank nonreimbursable advisory services and analytics projects (World Bank ASA and World Bank RAS), 380 IFC AS projects, and 1 IFC investment services (IFC IS) project that was approved between FY10 and FY20. 88 Manual Codification To analyze the selected projects’ intervention types and characteristics, IEG conducted the manual coding of the portfolio in two stages: » Issues Note. To expedite early understanding of the portfolio, the team initially reviewed 577 projects (a 35 percent sample), composed of 192 World Bank lending (52 percent of their identified portfolio), 225 World Bank ASA and World Bank RAS (25 percent), 159 IFC AS (42 percent), and 1 IFC IS (100 percent). The sample was random and stratified by institution, income level, and region. » Focused evaluation. The team manually reviewed project documentation of all 372 World Bank lending, 380 IFC AS, 1 IFC IS, 127 World Bank RAS iden- tified projects, and a 50 percent random sample of the World Bank ASA to understand DB projects’ intervention types and characteristics. In addition to the sample of 50 percent of the World Bank ASA projects, the team also analyzed the ones that were considered for the countries’ case studies. So, in total, we analyzed 57 percent of the World Bank ASA projects. While this systematic approach prioritized keyword searches of project ob- jectives, as well as indicators in project result matrices, the team also read projects’ documentation to include those whose objectives, components, and result matrices were informed by the DB report and those where DB had a Independent Evaluation Group World Bank Group    89 relevant influence on their rationale or motivation. The team followed a strict protocol, including a double-check routine to guarantee quality control. Supervised Learning Methods In collaboration with IEG’s methods advisory team, the evaluation com- plemented manual coding with automatic coding, adapting both shallow and deep learning techniques for classification. This was a pilot aimed at identifying World Bank lending projects that used DB in their rationale. This process was especially useful to identify qualifying projects that were excluded from initial human search (possible false negatives). The team used the following steps to identify those projects: » Initiation: The methods advisory team integrated relevant details from the underlying portfolio into a preliminary conceptual framework. The frame- work incorporated an aggregate description of the portfolio and assessed the composition of the input data to identify any potential challenges that might be encountered in content analysis. » Taxonomy: Building on insights from the learning sample already completed, a taxonomy was generated using a combination of both manual and automat- ic label recognition techniques, yielding a coding scheme for the classifica- tion of salient DB influence in the portfolio. » Automatic coding: This step integrated both shallow and deep learning tech- The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix A niques for classification. Shallow learning techniques (for example, n-gram analysis, term frequency—inverse document frequency) provided a sense of overarching patterns in the data, generating a broad assessment of rela- tive DB occurrences in the portfolio. Deep learning techniques were used to derive contextual meaning and add granularity to the coding scheme. Manual and automatic coding occurred iteratively along with intercoder reliability tests to improve the reliability and validity of the taxonomy. » Deployment: Once the taxonomy was sufficiently fine-tuned, it was deployed against the documents. Output from supervised learning was iteratively vali- dated to ensure consistency relative to the original taxonomy.1 Searches were conducted in structured text data available in the system (for example, PDO, component titles, indicators). » Validation: This process identified 226 potential World Bank lending DB-re- lated projects. The team manually validated project documentation of 107 projects to assess the classification of DB-related projects, representing a random sample of 40 percent plus the projects that were considered for the countries’ case of studies. World Bank Group Strategies Review IEG reviewed Forward Look: A Vision for the World Bank Group in 2030 and IFC 3.0 to identify if they referenced the DB report or indicators to any extent. The team also examined the World Bank Annual Report 2019 and IFC Annual Report 2019. Two main findings arose from this review: 90 1. Although DB is not explicitly embraced in Bank Group strategies, regula- tory reform is. 2. The Bank Group often focuses resources on improving countries’ business environment using DB as a metric. Advisory Panel An ad hoc advisory panel, comprising two internal and two external experts, provided guidance at two different stages of the evaluation. Review of Prior Evaluations The team reviewed independent reviews and evaluations which IEG and the World Bank have either conducted or commissioned in the past 12 years to assess DB indicators and their use. The prior work includes IEG’s 2008 evaluation Doing Business: An Independent Evaluation—Taking the Measure of the World Bank-IFC Doing Business Indicators and IEG’s 2015 evaluation Investment Climate Reforms: An Independent Evaluation of World Bank Group Support to Reforms of Business Regulations. IEG’s 2016 learning product, Sup- porting Transformational Change for Poverty Reduction and Shared Prosperity, also treated both catalytic qualities and limitations of DB. In addition, IEG’s 2017 evaluation Data for Development noted the influence and wide use of DB, and IEG’s 2019 evaluation Grow with the Flow: An Independent Evaluation Independent Evaluation Group World Bank Group    91 of World Bank Group Support to Facilitating Trade 2006–17 reviewed the DB trading across borders indicator. In the last decade, the World Bank commis- sioned two independent reviews of DB: the 2013 Independent Panel Review of the Doing Business Report (produced by a panel selected by the president of the World Bank Group), and the 2018 external audit On the Integrity of the “Ease of Doing Business” Indicators: Final Report. On the one hand, these reviews of DB recommended the continuation of the DB indicators and found substantial benefits in motivating country policy reform. On the other hand, each study also identified key areas of attention and action required to strengthen the DB indicators or their use. Desk Review of the Doing Business Literature IEG conducted a desk review of 426 articles from 100 leading academic jour- nals as an initial stocktaking of literature-based evidence. The database of articles had been previously identified and collected by the DB team and were shared with IEG. The DB literature database contains a research literature with abundant coverage of a few DB indicators (starting a business, protecting minority investors, and trading across borders) and much sparser coverage of others (for example, dealing with construction permits and getting electrici- ty). The desk literature review did not assess the methodological rigor of the published articles. The review found areas with strong evidence (as indicated The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix A by confirmation in multiple articles) on the association of selected indicators with development and economic outcomes for businesses and economies. Yet there were substantial gaps around some indicators. Much of this literature finds significant associations of DB indicator measures with outcomes, with- out necessarily establishing causation. Some of the literature casts doubt on the association of the measure either with the underlying area it is tracking (for example, construction permits) or with outcomes. Structured Literature Review IEG commissioned and supervised a structured literature review regarding the effects of business regulatory reforms on different outcomes by DB area (appendix F). The objective of this review was to minimize researcher bias by collecting and appraising all available research that has been identified us- ing an explicit literature search strategy and meeting prespecified eligibility criteria. The structured literature review followed the PICOS(LY) approach to specify the characteristics of eligible studies, outlining includable interven- tions, comparison groups, outcomes, study designs, and the written language of the study. Identifying a comprehensive list of studies for this review was challenging because of the scope and breadth of the various aspects of the regulatory environment covered by the DB project. The search required looking for evidence across a wide base of literature in the DB database, in the 3ie database, and through citation tracking using Google Scholar’s electronic citation tracking system. Screening at title and abstract removed studies with no clear relevance to the topic being reviewed. In addition, full- 92 text screening removed studies that did not specifically meet the inclusion criteria. Of the 1,894 manuscripts screened at full text, 103 records met the inclusion criteria. Review of Doing Business Report Affirmative Claims The team reviewed DB reports published between 2010 and 2020 to identify claims about the impact of DB-related reforms and their channels of influ- ence. The review aimed to understand the areas and types of reforms, and how they were linked to different outcomes. The identification of relevant claims followed a combination of manual codification and supervised learn- ing methods that were implemented in collaboration with IEG’s methods advisory team and included the following steps: » Taxonomy and initial identification: Building on insights from the evalua- tion theory of change and automatic label recognition techniques, the team generated a search taxonomy (keywords/phrases) that was used to identify sentences that used relevant words/phrases and were therefore likely to be making potential claims. This initial review identified 4,085 potentially rele- vant sentences in the 11 DB reports. » Learning sample: In order to refine the initial selection, the team created a learning sample or training data set based on manual screening of a 3 percent random sample (151 paragraphs) stratified by report and taxonomy term, to Independent Evaluation Group World Bank Group    93 classify whether a sentence was relevant or not. » Classification models: Using the learning sample, the team ensembled three text classification models (logistic regression, support vector machine, and multilayer perceptron) to estimate the probability of relevance for each of the 4,085 sentences. This exercise assigned 184 sentences a probability greater than an established threshold of 79 percent in all of the three models. » Manual screening: The team manually reviewed the 184 sentences and their adjoining sentences for reference. Within the paragraphs identified, the team eliminated those that did not have coherent framing of reform-related effects and those that were duplicates. Then, the team manually looked for mentions of mechanisms between reforms and their immediate outcomes, intermediate outcomes, and impacts. Indicator-Specific Deep Dives Indicator-specific deep dives provided an opportunity for the evaluation to study DB areas in a structured and focused manner. The deep dive methodol- ogy, guided by a common template, included: (i) a focused literature review, (ii) a review of relevant reforms supported by the Bank Group through its portfolio and a review of the treatment of the DB area in country-related documents, (iii) a cross-case-study analysis, and (iv) interviews with expert practitioners. The selection of deep dives was based on prevalence of areas in recorded projects, country reforms, and literature. Each of them delved into the DB indicator and subindicator features and examined construct and The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix A content validity (the degree to which the index or indicators measure the concept they intend to and the extent to which the index or indicators are representative of the area they aim to measure). Deep dives also provided detail on the relevant portfolio’s design features and drew on evidence from portfolio review and analysis on relevance and effectiveness, including fac- tors that were associated with success and failure. In addition, they included evidence from the structured literature review on the corresponding DB area. Deep dives also drew from the draft case studies and considered the role of stakeholders (other than the Bank Group) at the country of global level. Semistructured Interviews Semistructured interviews were conducted with 20 subject matter experts, managers, and practitioners within the World Bank Group and external bodies, including governments and academic institutions. At an early stage of the evaluation, an ad hoc panel assisted in identifying key issues and challenges to better understand the underlying theory, and to develop a set of preliminary hypotheses. The same panel was later consulted on emerg- ing findings. The interviews with Bank Group staff engaged in DB projects helped the team to understand institutional priorities, program features, achievement of objectives, and lessons derived from the experiences. They also helped to gauge stakeholders’ perspectives on the accuracy and useful- ness of DB for identifying reform priorities, guiding the design of reforms, and monitoring reform outcomes. The evaluation team developed an inter- view guide to ensure key questions were asked consistently across interviews 94 while maintaining the flexibility needed to follow response trajectories that might not adhere to the guide. Statistical and Econometric Analysis The evaluation applied statistical and econometric methods to examine the relevance and effectiveness of DB indicators. The team provided evidence on the reliability of the DB indicators by assessing (i) whether DB indicators ac- curately measured specific aspects of the regulatory environment, (ii) wheth- er the DB aggregate score was able to measure the quality of client countries’ regulatory environments, and (iii) whether DB indicators accurately iden- tified the right regulatory policy priorities. To conduct this analysis, IEG collected several measures of the regulatory environment from different sources to correlate them with the corresponding DB indicators. In addition, the team estimated a multivariate logistic regression to relate DB-informed interventions’ outcomes to possible predictors of reform success, including factors of success and country-level characteristics. Further efforts to apply econometrics to relate DB-measured country reforms to development out- comes did not prove conclusive. Country Case Studies The evaluation conducted 10 country case studies in purposively selected countries. Case selection reflected a diversity of country conditions and con- Independent Evaluation Group World Bank Group    95 texts (including regional diversity, income level, and International Develop- ment Association and fragility, conflict, and violence [FCV] classification). The country selection criteria included both high and low achievement of DB reforms—whether supported or not by the Bank Group—and high or low level of supportive Bank Group projects. The sample represented all regions and income categories of countries, as well as FCV and non-FCV countries. The evaluation team also consulted experts on which countries offered the richest opportunities for learning. The evaluation adopted a template for data collection and followed standard protocols to facilitate comparison across case studies. These protocols included (i) a review of the country’s business environment and development challenges; (ii) a review of country experience with DB; (iii) a review of Bank Group country strategies, diagnostics, and analytical works; (iv) a review of DB-related reforms supported or not by the Bank Group; (v) structured interviews with stakeholders (government, multilat- eral development banks, private sector, nongovernmental organizations, academics, and so on); and (vi) the extent to which DB-related reforms (supported or not by the Bank Group) achieved their desired outcomes and contributed to improved economic and social outcomes. Cases involved re- mote field-based assessments and aimed to identify to what extent the Bank Group–supported reforms and country-led reform efforts were effective and how and why specific reforms or reform interventions were or were not suc- cessful in delivering the intended results. Two team workshops strengthened The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix A learning across case studies and deep dives. Country-Level Reviews of Strategies and Diagnostics The evaluation conducted a series of systematic document reviews for a sample of countries to complement the evaluation’s portfolio review. The evaluation carried out a systematic review of all the Bank Group Country Partnership Frameworks approved after FY15 to better understand the level of alignment and coherence of Bank Group country-level strategies and DB-related concerns. A similar review was carried out for a random sample stratified by income level, region, and FCV status for those countries which had been subject to Systematic Country Diagnostics and all 18 of the pub- lished Country Private Sector Diagnostics available at the time of the review. A categorical array was developed to systematically assess evaluation ques- tions across strategy documents and diagnostics. Review of Relevant Country Program Evaluations and Completion and Learning Report Reviews IEG complemented the Country Partnership Framework review by analyz- ing the most recent country strategy CLRRs to learn the extent to which the World Bank’s country strategy objectives were informed or motivated by DB, whether they achieved their business environment reform outcomes, and the relevant lessons from the reviews. To make it comparable, the team reviewed the latest CLRRs with their corresponding Country Assistance Strategies or 96 Country Partnership Strategies of the same countries as in the Country Part- nership Framework revision. The final review included 61 countries with an available CLRR. The team also reviewed some Country Program Evaluations to extract examples of DB use. Review of International Finance Corporation Advisory Services and Project Performance Assessment Report Chapeau Report IEG conducted deep dives regarding 50 evaluation notes for all IFC AS–eval- uated projects and seven available Project Performance Assessment Reports, which in both cases were drawn from the identified population of DB-in- formed projects in the portfolio review and analysis. This analysis aimed to enrich the understanding of the factors of success and failure and extract lessons learned. Design Matrix Table A.6 lists evaluation questions with the evaluation design. The number of check marks represents the strength of the method to answer the questions. Table A.6. Evaluation Questions and Methods Applied 1. Doing the Right Things: Is DB doing the right things in terms of specific indi- Independent Evaluation Group World Bank Group    97 cators and country contexts? What is the relevance of DB indicators to a. The business environment priori- ✓✓✓ ✓✓ ✓✓ ✓✓ ✓✓✓ ✓✓✓ ✓✓✓ ties of client countries? b. The substantive dimensions within ✓✓✓ ✓✓ ✓✓✓ ✓✓✓ ✓✓ ✓✓✓ each indicator’s focus area? c. World Bank Group strategic and ✓✓✓ ✓✓ ✓✓ ✓ ✓✓✓ ✓✓✓ ✓ operational priorities? (continued) 2. Doing Things Right: Is DB being used by clients and the Bank Group in ways to achieve the best effect for business environment reform? a. Is DB effectively achieving desired ✓✓✓ ✓✓✓ ✓✓ ✓✓ ✓✓ ✓✓ ✓✓ outcomes? b. Is DB being used by clients and ✓✓✓ ✓✓ ✓✓ ✓✓ ✓✓ ✓✓✓ ✓ the Bank Group to achieve the best outcomes for business environment reform (such as entry, investment, and employment) subject to broader policy priorities (such as public safe- ty, equity, and environment)? Source: Independent Evaluation Group. The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix A Note: The number of check marks indicates the strength of the method to answer the question, from one check mark suggesting that the method provided some data to answer the evaluation question, to three check marks suggesting that the method provided a great deal of data to answer the evaluation question. DB = Doing Business. Design Limitations Notwithstanding these steps, the evaluation methodologies had limitations related to gaps and inconsistencies in the literature and country social and economic data sets; the limited number of relevant evaluated projects (and even smaller number of evaluated unsuccessful projects) and country pro- grams with DB relevance (especially in light of the lack of IEG-validated evalu- ations of ASA); the limited ability of a limited number of case studies and deep dives to represent the universe; limited data on intermediate outcomes and sustainability of reforms and difficulties in establishing attribution to specific reforms; and difficulties in conducting fieldwork (even remotely) imposed by COVID-19. There were also some false negatives or false positives in identify- ing relevant literature, project activities, and reforms due to the multifaceted nature of areas covered by some DB indicators, lack of standard terminology, and the sometimes indirect nature of influence. Establishing causal connec- tions between DB indicators and observed reforms, and between reforms and observed outcomes, was complicated by the likelihood of multiple causal factors and the potential for omitted variables. 98 1  Among others, unsupervised learning methods, such as word embeddings and machine-gen- erated synonyms from Google’s Word2Vec or Facebook’s fastText, can be used. Independent Evaluation Group World Bank Group    99 Appendix B. Portfolio Review Relevance This portfolio review aims to provide an overview of World Bank Group projects that referenced Doing Business (DB), either as indicators or as rele- vant dimensions in their rationale, objectives, or result matrices, approved between fiscal years 2010 and 2020 (FY10–20). This support was channeled almost exclusively through two Bank Group institutions: the World Bank and the International Finance Corporation (IFC). In the case of the World The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix B Bank, its support included lending operations (World Bank lending), as well as reimbursable and nonreimbursable advisory services and analytics (ASA), whereas IFC support was delivered through advisory services (IFC AS). The Independent Evaluation Group (IEG) estimates there were 269 DB-informed lending projects and 407 advisory services projects approved from 2010 through 2020, with a total commitment value for DB activities of US$15.5 billion. Of these 676 projects, 137 were evaluated by IEG. Most of the sup- port was provided through advisory services (60 percent), of which the World Bank provided 55 percent while the IFC delivered 45 percent. Regarding financing projects, the World Bank accounted for all DB lending projects and around 96 percent of volume in dollar value when accounting for total proj- ect volume ($14.9 billion; table B.1). 100 Table B.1. Summary of Doing Business–Related Portfolios, Approved FY10–20 (projection) Volume/ Projects Interventions Commitments (US$, Institution (no.) (%) (no.) (%) millions) (%) World Bank 269 40 517 43 14,853 96 lending Lending 269 40 517 43 14,853 96 subtotal World Bank 165 24 173 14 379 2 ASA World Bank 58 9 81 7 20 0.1 RAS IFC AS 184 27 428 36 287 2 Advisory 407 60 682 57 686 4 services subtotal Total 676 100 1,199 100 15,539 100 Source: Independent Evaluation Group, portfolio review analysis. Note: All information is projected based on the population and sample sizes. Specifically, the results were multiplied by a factor of 226/107 for World Bank lending projects identified by machine learning, and 789/452 for World Bank ASA projects. Volume, commitment, or funds managed were identified or estimated according to what was allocated to Doing Business–related interventions. If the amount was not explicitly stated, it was estimated based on the number of components, subcompo- nents, or activities. ASA = advisory services and analytics; IFC AS = International Finance Corporation advisory services; RAS = reimbursable advisory services. Independent Evaluation Group World Bank Group    101  istribution of Doing Business Projects by Income Level, Figure B.1. D FY10–20 (projection) a. World Bank Group projects by income level and institution The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix B b. Share of projects by institution and income level Source: Independent Evaluation Group, portfolio review analysis. Note: Excludes 70 regional projects and projects in income-unclassified countries. All information is projected based on the population and sample sizes. Specifically, the results were multiplied by a factor of 226/107 for World Bank lending projects identified by machine learning and 789/452 for World Bank ASA projects. ASA = advisory services and analytics; IFC AS = International Finance Corporation advisory services; RAS = reimbursable advisory services. 102  istribution of Doing Business Projects by Region, FY10–20 Figure B.2. D (projection) Source: Independent Evaluation Group, portfolio review analysis. Note: All information is projected based on the population and sample sizes. Specifically, the results were multiplied by a factor of 226/107 for World Bank lending projects identified by machine learning and 789/452 for World Bank ASA projects. ASA = advisory services and analytics; EAP = East Asia and Pacific; ECA = Europe and Central Asia; IFC AS = International Finance Corporation advisory services; LAC = Latin America and the Caribbean; MENA = Middle East and North Africa; OTH = other; RAS = reim- bursable advisory services; RGN = regional; SAR = South Asia; SSA = Sub-Saharan Africa. Independent Evaluation Group World Bank Group    103 104 The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix B  upported Countries, Projects, and Interventions by Region, FY10–20 (projection) Table B.2. S Financing Advisory Services* Volume Volume Projects Avg. projects (US$, Projects Avg. projects (US$, Region Countries (no.) per country million) Countries (no.) per country million) SSA 39 113 2.9 2,350 43 116 2.7 434 ECA 19 51 2.7 2,236 23 60 2.6 36 LAC 15 24 1.6 3,071 18 44 2.5 28 EAP 10 28 2.8 1,549 17 48 2.8 43 MENA 6 27 4.5 3,926 18 43 2.4 34 SAR 8 20 2.4 1,314 7 33 4.8 55 RGN   7   407   39   47 OTH           23   9 Total 97 269 2.8 14,853 126 407 3.2 686 Source: Independent Evaluation Group, portfolio review analysis. Note: All information is projected based on the population and sample sizes. Specifically, the results were multiplied by a factor of 226/107 for World Bank lending projects identified by machine learning and 789/452 for World Bank advisory services and analytics projects. Volume, commitment, or funds managed were identified or estimated according to what was allocated to Doing Business–related interventions. If the amount was not explicitly stated, it was estimated based on the number of components, subcomponents, or activities. EAP = East Asia and Pacific; ECA = Europe and Central Asia; LAC = Latin America and the Caribbean; MENA = Middle East and North Africa; OTH = other; RGN = regional; SAR = South Asia; SSA = Sub-Saharan Africa. * Advisory services include International Finance Corporation advisory services and World Bank advisory services and analytics projects.  istribution of Supported Doing Business Areas, FY10–20 Figure B.3. D (projection) Source: Independent Evaluation Group, portfolio review analysis. Note: All information is projected based on the population and sample sizes. Specifically, the results were multiplied by a factor of 226/107 for World Bank lending projects identified by machine learning and 789/452 for World Bank ASA projects. Projects can be counted more than once if they support more than one business area. ASA = advisory services and analytics; IFC AS = International Finance Corporation advisory services; RAS = reimbursable advisory services.  se of Doing Business Reports in Projects, FY10–20 Figure B.4. U (projection) Independent Evaluation Group World Bank Group    105 Source: Independent Evaluation Group, portfolio review analysis. Note: All information is projected based on the population and sample sizes. Specifically, the results were multiplied by a factor of 226/107 for World Bank lending projects identified by machine learning, and 789/452 for World Bank ASA projects. ASA = advisory services and analytics; DB = Doing Business; IFC-AS = International Finance Corporation advisory services; RAS = reimbursable advisory services.  istribution of Doing Business Intervention Types, FY10–20 Figure B.5. D (projection) The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix B Source: Independent Evaluation Group, portfolio review analysis. Note: All information is projected based on the population and sample sizes. Specifically, the results were multiplied by a factor of 226/107 for World Bank lending projects identified by machine learning and 789/452 for World Bank ASA projects. ASA = advisory services and analytics; IFC-AS = International Finance Corporation advisory services; RAS = reimbursable advisory services. Effectiveness at the Intervention Level This section analyzes the achievements of Bank Group DB-relevant inter- ventions for the 137 evaluated projects. IEG assessed the accomplishment of project objectives (with IEG-validated ratings and data available at the individual intervention level).1 The evaluated projects included 87 World Bank lending projects and 50 IFC AS, in which 303 interventions were iden- tified. However, for 12 interventions no relevant data were provided on their effectiveness; therefore, the denominator for the calculations reflected in the figures below is 291 (table B.3).  istribution of Evaluated Doing Business Projects and Inter- Table B.3. D ventions by Institution Interventions Projects Institution Number % Number % World Bank lending 157 54 87 64 IFC advisory services 134 46 50 36 Total 291 100 137 100 106 Source: Independent Evaluation Group, portfolio review and analysis. Note: Eight World Bank lending and four IFC advisory services interventions do not have data about their effectiveness. IFC = International Finance Corporation.  uccess Rate of Doing Business–Related Interventions, Figure B.6. S FY10–20 a. By institution* b. By income level Independent Evaluation Group World Bank Group    107 c. By Region The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix B Source: Independent Evaluation Group, portfolio review analysis. Note: Panels a and c are based on 291 interventions (which excludes 12 interventions with no data regarding their intervention outcomes), and panel b is based on 276 interventions (which excludes 27 interventions part of regional projects or unclassified countries/territories or with no data regarding their intervention outcomes). Success rate is defined as the proportion of interventions that achieved or mostly achieved their intervention outcomes. DPL = development policy lending; EAP = East Asia and Pacific; ECA = Europe and Central Asia; IFC AS = International Finance Corporation advisory services; IPF = investment project financing; LAC = Latin America and the Caribbean; MENA = Middle East and North Africa; RGN = regional; SAR = South Asia; SIL = specific investment loans; SSA = Sub-Saharan Africa. * Program-for-results financing is not reported because only one project was identified under that category. 108  uccess Rate of Doing Business–Related Interventions by Figure B.7. S Business Area, FY10–20 Source: Independent Evaluation Group, portfolio review analysis. Note: Figure is based on 291 interventions (which excludes 12 interventions with no data about interven- tion outcomes). Interventions may be counted more than once since they can support multiple busi- ness areas. Success rate is defined as the proportion of interventions that achieved or mostly achieved their intervention outcomes. * n < 5 interventions. Independent Evaluation Group World Bank Group    109 Table B.4. Success Rate of Doing Business–Related Interventions by Business Area and Institution, FY10–20 World Bank Lending IFC AS Business Area Number % Success Number % Success Ease of doing business 21 90 17 88 Trading across borders 23 87 16 75 Starting a business 31 90 29 66 Getting credit 14 79 30 90 Dealing with construction permits 14 93 19 84 Paying taxes 15 93 16 81 Registering property 9 78 10 50 The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix B Enforcing contracts 9 56 11 73 Resolving insolvency 6 33 9 67 Getting electricity 5 100 1 0 Protecting minority investors 6 100 1 100 Contracting with the government 1 100 0 n.a. Employing workers 3 100 0 n.a. Total 157 85 159 78 Source: Independent Evaluation Group, portfolio review analysis. Note: Based on 291 interventions (which excludes 12 interventions with no data about intervention out- comes). Interventions may be counted more than once since they can support multiple business areas. Success rate is defined as the proportion of interventions that achieved or mostly achieved their inter- vention outcomes. IFC AS = International Finance Corporation advisory services; n.a. = not applicable.  uccess Rate of Doing Business–Related Interventions, Figure B.8. S FY10–20 Source: Independent Evaluation Group, portfolio review analysis. 110 Note: Figure is based on 291 interventions (which excludes 12 interventions with no data about inter- vention outcomes). Success rate is defined as the proportion of interventions that achieved or mostly achieved their intervention outcomes. * n < 5 interventions.  uccess Rate of Doing Business–Related Interventions by Type Table B.5. S and Institution, FY10–20 World Bank Lending IFC AS Intervention Type (no.) (% Success) (no.) (% Success) Improve or build infrastruc- 2 100 0 n.a. ture Diagnostic 0 n.a. 2 100 Enhance interoperability of 1 100 1 100 processes or data-sharing Raising support for and 0 n.a. 9 100 awareness of reform Set up or reform agencies 7 57 4 50 Business environment strate- 2 100 9 78 gies or policies Support the use of electronic 16 81 14 71 systems or automation Capacity building and training 11 91 25 80 Reengineering process 44 84 30 77 Improve business laws or 74 88 40 75 regulation Total 157 85 134 78 Source: Independent Evaluation Group, portfolio review analysis. Note: Based on 291 interventions (which excludes 12 interventions with no data about intervention out- comes). IFC AS = International Finance Corporation advisory services; n.a. = not applicable. Table B.6. S  uccess Rate of Doing Business–Related Interventions by Independent Evaluation Group World Bank Group    111 Type of Use of the Doing Business Report, FY10–20 Use of DB Report / Indicators Number % Success As justification for project 183 82 As project indicator 142 80 As project objective 50 72 As support to generate DB report or to update indicators 6 67 Source: Independent Evaluation Group, portfolio review analysis. Note: Based on 291 interventions (which excludes 12 interventions with no data regarding their inter- vention outcomes). Interventions may be counted more than once since they can use DB reports in multiple ways. DB = Doing Business.  uccess Rate of Doing Business–Related Immediate Out- Figure B.9. S comes, FY10–20 The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix B Source: Independent Evaluation Group, portfolio review analysis. Note: Figure is based on 262 interventions (which excludes 42 interventions with no data regarding their immediate outcomes). Interventions may be counted more than once since they can have multiple immediate outcomes. Success rate is defined as the proportion of interventions that achieved or mostly achieved their intervention outcomes. * Denotes n < 5 interventions.  uccess Rate of Doing Business–Related Intermediate Out- Figure B.10. S comes, FY10–20 Source: Independent Evaluation Group, portfolio review analysis. Note: Figure is based on 150 interventions (excluding 153 interventions with no data regarding their immediate outcomes). Interventions may be counted more than once since they can have multiple in- termediate outcomes. Success rate is defined as the proportion of interventions that achieved or mostly 112 achieved their intermediate outcomes. * Denotes n < 5 interventions. Effectiveness at the Project Level IEG’s analysis indicates that components informed by DB were more suc- cessful on average than the overall projects of which they were components. For World Bank lending, while DB-informed components were 85 percent successful, DB-informed projects were only 70 percent successful. For IFC AS, while DB-informed components were 78 percent successful, DB-in- formed projects were 54 percent successful (figure B.11). Figure B.11. Success Rate of Doing Business–Informed Projects and In- terventions, FY10–20 a. Projects by institution* b. Interventions by institution* Independent Evaluation Group World Bank Group    113 Source: Independent Evaluation Group, portfolio review analysis. Note: Panel a is based on 137 evaluated projects; panel b is based on 291 interventions (which excludes 12 interventions with no data regarding their intervention outcomes). Success rate is defined as the proportion of projects or interventions that achieved or mostly achieved their intervention outcomes. DPL = development policy lending; IFC AS = International Finance Corporation advisory services; IPF = investment project financing; SIL = specific investment loans. * Program-for-results financing is not reported as only one project was identified under that category. Learning about Factors of Success and Failure IEG’s review of evaluated projects indicate there are 696 factors to which project success or failure were attributed, most within the Bank Group’s con- trol. These factors could be identified at the project level (not at the inter- vention level). Additionally, these factors were tagged as adequate (386) or inadequate (310; figure B.12).  xternal and Internal Factors Influencing Outcomes by Type Figure B.12. E of Impact, FY10–20 a. Type of factors The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix B b. Distribution of factors Source: Independent Evaluation Group, portfolio review analysis. Note: The figure shows the factors (696) identified for the 137 evaluated projects. Projects may be counted more than once since they can have multiple factors that affected their outcomes. M&E = 114 monitoring and evaluation.  xternal and Internal Factors Influencing Outcomes, by Use Figure B.13. E of Doing Business, FY10–20 a. Adequate b. Inadequate Independent Evaluation Group World Bank Group    115 Source: Independent Evaluation Group, portfolio review analysis. Note: Factors were identified at the project level, and one project may use the DB report in multiple ways. For this reason, factors may be counted more than once. DB = Doing Business; M&E = monitoring and evaluation. Figure B.14. World Bank Group Coordination, FY10–20 The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix B Source: Independent Evaluation Group, portfolio review analysis. Note: WBG = World Bank Group. Effectiveness and Learning about Factors of Success and Failure in World Bank Advisory Services and Analytics Projects Unlike World Bank lending and IFC AS, the framework for World Bank ASA self-evaluation was neither agreed to nor validated by IEG. Furthermore, the potential biases of unvalidated self-evaluation are evident from the overwhelming reported effectiveness rate of reimbursable advisory services (RAS) of 96 percent (figure B.15). Yet self-evaluation can be useful for capturing factors of ASA that either contributed positively to or detracted from project success. One key distinc- tion is that ASA is more subject to the influence of external factors, led by political economy and agency coordination issues, client commitment, and client capacity. Quality at entry was more influential when it was absent, often centered around the realism of the timetable for reform, the adapta- tion of the support to the client, and, in particular for RAS clients, excessive design complexity. Project supervision played a strong positive role, partic- ularly when there was effective coordination with partners, proactive client engagement, flexible implementation, and, in particular for RAS clients, proper team composition (figure B.16 and table B.7). 116 IEG extracted a 32 percent random sample (50 projects), stratified by region, income level, and year, from the 156 identified projects. Of the 50 sampled projects, the team could not identify factors of success/failure in 38 percent (19) of them. ASA projects represent 48 percent of the 31 remaining projects (15), while RAS projects represent 52 percent (16). Of the 31 remaining proj- ects, IEG identified 82 factors. Figure B.15. Overall Development Objective Effectiveness a. By type of advisory service b. By Doing Business use Independent Evaluation Group World Bank Group    117 Source: Independent Evaluation Group, portfolio review analysis. Note: IEG identified 156 ASA projects (unweighted). Of those, 127 (81 percent) were evaluated according to completion summary reports. Two projects did not report an overall development objective rate; nevertheless, other indicators were rated. One project may use the Doing Business report in multiple ways. For this reason, projects may be counted more than once in panel b. Highly effective = Excep- tionally effective + Very effective + Fully satisfactory + Fully achieved; Effective = Effective + Yes + Largely achieved + Satisfactory; Moderately effective = Moderately effective + Moderately satisfactory + Partially; Slightly effective = Slightly effective. AAA = advisory services and analytics; RAS = reimbursable advisory services; WB = World Bank. Figure B.16. External and Internal Factors by Type of Advisory Service a. Adequate The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix B b. Inadequate Source: Independent Evaluation Group, portfolio review analysis. Note: AAA = advisory services and analytics; RAS = reimbursable advisory services. 118 Table B.7. External and Internal Factors Adequate, % Inadequate, % Factor (n = 56) (n = 26) Quality at entry Suited to client capacity 0 8 Choice of instrument 0 4 Enabling environment 2 0 analysis Design complexity 2 12 Timetable realism 5 19 Project Team composition 11 4 supervision Flexibility of implementation 13 0 Proactive client engage- 20 0 ment and follow-up Effective coordination with 21 0 partners, donors, imple- menters M&E Design 0 8 considerations Implementation 2 8 External factors Other 0 4 Agency coordination and 2 15 political economy Source: Independent Evaluation Group, portfolio review analysis. Note: M&E = monitoring and evaluation. Independent Evaluation Group World Bank Group    119 1  Information was drawn from Implementation Completion and Results Report Reviews, Ex- panded Project Supervision Reports, and evaluative notes. The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix B 120 Appendix C. Case Studies Methodological Note The 10 country case studies were developed through document desk reviews, online interviews with headquarters staff, and online (and in-person) local interviews with World Bank Group staff, government staff, and civil society members (entrepreneurs and their representatives, academics, and others). COVID-19 pandemic conditions mandated that most fieldwork was conduct- ed remotely. The case studies were completed in May 2021 and pertain to experience up to that time. Following the templates used for the evaluation case studies, each background note is summarized in this appendix to ad- dress, whenever possible, the following: » Doing the right things: Is Doing Business (DB) doing the right things in terms of indicators and the country context? » Country context and reform priorities » Country experience with DB » Bank Group’s role and relevance » Doing things right: Is DB being used by clients and the Bank Group in ways to Independent Evaluation Group World Bank Group    121 achieve the best effect for business environment reform? » Effectiveness » Factors of success and failure » Internal » External Cross-cutting findings from the case studies and deep dives are summarized in table D.1 at the end of appendix D. Afghanistan » Doing the right things: Is DB doing the right things in terms of indicators and the country context? Afghanistan is a nation with severe development challenges. Persistent high poverty rates, high levels of inequality, and low economic growth are some of the main factors influencing the country’s economic and social outcomes. Its status as a fragility, conflict, and violence country was a binding constraint throughout the evaluation period of fiscal years 2010–20 (FY10–20). These and other development challenges have been amplified by persistent and growing insecurity and uncertainty, and by declining aid. International de- The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix C velopment assistance contributed to growth and jobs but did not contribute to raising productivity. It resulted in a problematic business environment, with issues that go beyond the Doing Business indicators (DBIs) but with some issues that are captured by them. With regard to its rankings, the coun- try is among the lowest ranked in the world, lagging the regional average in 9 of the 10 regulatory areas measured by the DBIs. There has been a general alignment of DB-informed reforms with Afghan- istan’s national priorities. According to Bank Group documents as well as non–Bank Group information, national priority programs are both directly and indirectly influenced by DBIs. Partner government ministries view the DB indicators as references and benchmarks while implementing newly introduced reforms. Many of the DB-informed reforms are adapted appropri- ately to the country context by prioritizing those objectives that are possible given available resources and capacity, but there are exceptions. An example involves starting a business: An entrepreneur must go through three proce- dures that take seven days on average, but women face an additional proce- dure of obtaining their husband’s approval to leave home, a legal reality that many businesswomen consider to be misinterpreted and outdated. Stakeholders use the DBIs as a reputable source, discussion driver, indicator, and objective. Many evaluation documents by outside entities (for example, Asian Development Bank) also reference DBIs as a reputable source. Com- plementary data and analytic work enhance the relevance of DBIs in design- ing or selecting business environment reforms and in following up on them. 122 DBIs are often the root source or discussion driver behind reforms and are used in the design phase of reforms as background documents for gap analy- ses and baseline assessments. Limits to the DBIs’ relevance emerge in considering the limited improve- ments have yielded for investment. For instance, the country climbed 16 places in 2018’s DB ranking. As mentioned in the United Nations Develop- ment Programme country report (UNDP Afghanistan 2021), Bangladesh was ranked nine places below Afghanistan and scored 20 percent lower. Despite that, Bangladesh has a more dynamic economy and is more successful in at- tracting investment. This example raises important points on relevance and effectiveness of DBI-related reforms, as well as concerns about the empirical evidence for them. It shows that numerical assessments can at times be mis- leading regarding the status and potential of an economy. » Doing things right: Is DB being used by clients and the Bank Group in ways to achieve the best effect for business environment reform? DBI-related reforms (both supported by the Bank Group and not) contributed to improvements in country positions in some indicator rankings during the evaluation period, but others did not improve. Since 2015, the government has made efforts to create a more dynamic environment for private sector development (mostly composed of small and medium enterprises, SMEs), but policy formulation and implementation still require further attention. Independent Evaluation Group World Bank Group    123 The government made significant efforts over the past years to reduce the cost to register a company (starting a business was the government’s focus area). Private stakeholders indicated that since a recent decrease in business registration fees, the burden was overcome. Indeed, that is reflected in the country’s 52nd ranking in the indicator (DB2020). Nonetheless, the process to register a business is still comparatively costly, as all businesses must pay an annual license fee corresponding to 82 percent of income per capita (much higher than the regional average of 21 percent). Clients and the Bank Group using DB have been able to achieve some im- proved outcomes for business environment reform, but much remains to be done. The limited measurable outcomes in such dimensions as formal investment and employment are cautionary. Despite International Finance Corporation (IFC) advice to PriSEC (Executive Committee for the Private Sector) on paying taxes, technical and institutional support on trading across borders, and support on getting credit, there is no firm evidence that all this made a difference on the ground (through, for example, businesses finding it easier to pay taxes, trade, or get credit; an increase in taxes paid, imports, and exports; or loans to private firms). Reliable data availability would be needed to assess whether the work done has been effective. According to Bank Group data, which are only available until 2017, tax revenue as a per- centage of gross domestic product (GDP) increased from 2014 until then. As for trading across borders, there are not enough publicly available data that might show the impact of reforms on imports and exports. The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix C Lessons A monitoring and evaluation framework and identification of risks at ap- praisal has been perceived by many ministerial directorates as major factors that helped ensure specific DBI-informed reforms or interventions were successful. During the implementation phase, the framework has helped stakeholders to carry out activities in a timely manner. It was particularly effective when multiple ministries were involved in implementing a specific DBI-informed reform, reducing friction among them and with outside enti- ties including the Bank Group. Some external factors such as public sector institutional strength and sophistication, private sector capacity, and agen- cy coordination have proved to be primary challenges facing the successful implementation of DBI-informed reforms. For instance, activities under the starting a business indicator require implementing a full online busi- ness registration system, which is not realistic given the country’s private and public sector capacity. Governance capacity is limited in Afghanistan, perhaps indicating that a great deal of simplification and hand-holding is required in DBI-related reform. Lack of sufficient human resources is a notable gap in the government’s capacity to implement and sustain reforms and regulations. To address this issue in the short term, continuous capacity building of human resources is needed. In the long term, the focus should be on strengthening the educational system in the country to help raise a gen- eration with the necessary culture and technical skills to help bring about sustainable economic development. Stakeholders seem to perceive that the 124 current institutional arrangements in the Bank Group do try to separate indicator generation from project work, including reimbursable advisory services (RAS). There has thus not been any reported conflict of interest. Yet these undoubted achievements were made at the cost of some friction between different stakeholders. China » Doing the right things: Is DB doing the right things in terms of indicators and the country context? Reflecting its early rapid growth of preceding decades, China became the world’s second-largest economy, with great success with poverty reduction and virtually all Millennium Development Goals. China’s GDP growth re- mained spectacular over the start of the evaluation period in 2010 (continu- ing from 1983), but growth slowed gradually after 2012, decelerating from 7.8 percent in 2013 to 6.6 percent in 2018, and further to 5.8 percent in 2019. Yet China’s market reforms remained incomplete, and poverty continued to be a major challenge. As early as in China’s 11th Five-Year Plan (2005–10), it was recognized that the country’s growth—led by a pattern of moves from agriculture to industry, followed by partial market reforms coupled with heavy state-owned enterprise (SOE) investment—would likely weaken. Chi- na’s 12th Five-Year Plan (2011–15) sought to address some of these issues, Independent Evaluation Group World Bank Group    125 with a new emphasis on services and energy efficiency, as well as attention to social imbalances. There is a clear evolution in how DB has been viewed by the government. The 2012 Country Partnership Strategy did not emphasize the acceleration of market reforms or the private sector. China’s 2018 Systematic Country Diagnostic (SCD) described the economy as rebalancing toward a new low- er-growth equilibrium with structural shifts, away from its combined focus on heavy-industrial investments and low-wage and energy-intensive man- ufacturing and construction toward consumption, and from manufacturing to services. A comprehensive reform agenda was laid out in the 13th Five- Year Plan (2016–20) to facilitate the economic transition. The 2019 Country Partnership Framework (CPF) for the period 2020–25 brought attention to fiscal, institutional, and governance issues and systematic market reform. It emphasized improving the environment for competition and private sector development and sustainable subnational fiscal management. The DB indicators have been of enormous influence in China since 2017. The country did seek rank improvement, but there grew to be a genuine and crit- ical appreciation of the value of DB as a diagnostic and benchmarking tool that would help cities within China to assess their performance and identify areas for change. City-wide reform plans were modeled on DB findings and recommendations. The Ministry of Finance mobilized staff at the highest levels in Beijing and Shanghai (the two cities tracked by DB). That ministry and World Bank China also guided uptake of some indicators in Guangzhou The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix C and Shenzhen. The World Bank’s subnational DB independently conducted a study of select indicators in Chongqing. The civil service displayed a critical and detailed knowledge of the indicators and their limitations. The National Development and Reform Commission began exploring possible Chinese indicators and applications outside the principal cities. Relevant areas were identified, but their formalization as actual indicators is not yet evident. The 2012 CPF did not emphasize the acceleration of market reforms or the private sector. However, the IFC had focused in the previous Country As- sistance Strategy on DB reforms. The 2006–11 Country Partnership Strat- egy included simplified business entry and regulation as an outcome and activities (covering 29 provinces). The different approach toward DB was influenced by a combination of forces. First, internal to China, there was a growth slowdown after the financial crisis, recognized low SOE productiv- ity, a high-level recognition of the need to find new drivers of growth, and a recognition of the need for a better-regulated business environment for more orderly private sector growth. DB’s potential for aiding this was begin- ning to be recognized, together with the value of an improvement in Chi- na’s DB rankings in the international arena. Second, external to China, the World Bank recognized the potential for new engagement in this area and elevated DB in country dialogue. A close relationship developed starting in 2017, initially with World Bank technical assistance and soon with a rapidly growing RAS program. The World Bank’s Beijing office provided significant support to China. World Bank experts produced high-quality RAS reports. 126 Thus, DB became a reform motivator in recent years, but pivotal support for the private sector had already begun years before. A comparison with a China enterprise survey undertaken by the World Bank around the start of the evaluation period, over a broad geographic sample, suggests that DB did not cover many issues perceived to be important to the enterprise sector at a micro level, such as access to finance. China’s Ministry of Finance officials point to areas in which the DB indica- tors may be intrinsically limited as tools of measurement, ill adapted to the Chinese context, outdated, or missing key elements. Efforts have been made to identify and include such areas in cities’ business plans even if not for- malized into measurable indicators. External investors do not use DB indica- tors to gauge the business environment. They launch their own surveys and use other statistical sources. They point to the lack of coverage of critical themes: a level playing field, whether by ownership (state/private, domestic/ foreign), size, and sector of operation, given the negative lists of areas where foreign participation is not permitted. Partners in aid point to other missing areas: law enforcement, an autonomous judiciary system, and intellectual property rights. Other experts highlighted labor market, financial sector or macroeconomic factors outside the scope of the DB indicators. Experts interviewed by IEG considered that DB does not capture many factors critical to China’s business environment. It is unclear if there is any correlation of China’s DB scores with other measures of business confidence, Independent Evaluation Group World Bank Group    127 such as the Organisation for Economic Co-operation and Development’s (OECD’s), or the World Economic Forum indexes. » Doing things right: Is DB being used by clients and the Bank Group in ways to achieve the best effect for business environment reform? Despite recent striking improvements in DB rankings, the generally ex- pressed view by expert observers is that the last five years have been an uneven process of reform of the business environment, with some reversals and unclear trends, alongside some areas such as starting a business, where there was some undoubted improvement. By contrast, in earlier years, when China was trailing in DB, foreign confidence was high and foreign invest- ment poured in. During an earlier period of rapid private sector growth up to 2014, improvements were not guided by DB. Some changes were achieved at a national level, though others remained in the realm of the city governments. Conferences were held with partici- pants from other cities to foster rollout. The message was spread through local offices of the National Development and Reform Commission. Perfor- mance evaluation incentives were also applied. The World Bank Country Office guided the uptake of some indicators in Guangzhou and Shenzhen. The World Bank’s subnational DB independently conducted a study of select indicators in Chongqing. The World Bank subnational DB of 2008 and the World Bank enterprise survey of 2012 point to significant regional differ- ences. Although only 30 reforms were counted in the DB database over the evaluation period, local governments claim many more—Beijing claims over The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix C 400 reform measures, Shanghai over 300, and Chongqing over 200. There is some preponderance of reforms with easy procedural simplification through one-stop shops, reducing time, fees, and costs. DB China reaped the digital dividends of connectivity, the internet, big data, and thus accessible infor- mation sharing across agencies and online procedures. Some changes that appear more significant are very recent and are yet to be assessed by DB. Authorities were aware of the changes that would influence rankings, even if this was not the only motivating factor. Lessons For World Bank country offices where DB is seeking to expand its traction, important factors are buy-in and overlap of the official agenda with DB. State resources, motivation, and competent officials are a part of this. Support from the World Bank, elevation of DB in the country’s own agenda, the provi- sion of technical assistance, building a RAS pipeline, and doing high-quality work are also critical. For DB to remain relevant at a deeper level, there is a need for micro change in the indicators, model cases, and respondents; more universally appropriate settings; and much more regional representation. This implies difficult trade-offs in terms of comparability across countries, costs, and comparisons over time. The loss of DB relevance has accelerated with sea changes in business environments thanks to big data and the inter- net. Moreover, DB is intrinsically susceptible to focus shift by highly trained officials to those areas of change that are low-hanging fruit. There is a need for DB to be aware of this. Even more fundamental for the relevance of DB is 128 the larger question of what it takes to create a competitive business envi- ronment and whether DB indicators can be used as a gauge for this. China illustrates that many other factors are needed to gauge business competi- tiveness even if DB measures are helpful in a limited sphere. If DB is to be used as a more universal measure of the business environment, it will need at a minimum an expansion of theme, incorporation of enterprise sentiment, and regional extension. The resulting product, however, may be unrecogniz- able as a part of DB today. Colombia » Doing the right things: Is DB doing the right things in terms of indicators and the country context? Colombia is an upper-middle-income country with a strong policy frame- work but deep-rooted challenges. Its economy is the fourth largest in Latin America as measured by GDP, and its population is over 50 million. Its un- even territorial development, an armed conflict that went on over 50 years, and a recent emergence of extractive industries have deep historical roots that condition the achievement of poverty eradication and shared prosperity in a sustainable manner. (SCD 2015) Since the mid-2000s, administrations have made business competitiveness and productivity growth an explicit policy objective, enjoying continuous political support. They were initial- Independent Evaluation Group World Bank Group    129 ly focused on reducing businesses’ transaction costs through simplifying procedures. Although with fading momentum until later in the decade, these types of reforms continued during 2010–20, and they were accompanied by a gradual shift toward strengthening regulatory institutions and processes aimed at promoting high-quality, rational regulations. More complex and expensive reforms of legal institutions, such as bankruptcy systems and in- vestor protection, were also carried out during the decade. DB was used as a reference and supporting tool to inform Colombia’s busi- ness environment diagnoses and as a monitoring and communication tool to strengthen its national and international image. Due to DBIs’ narrow scope, DB is monitored in combination with other sources (for example, World Economic Forum’s Global Competitiveness Report, International Institute for Management Development’s World Competitiveness Yearbook). The government has monitored and socialized DB-measured progress in its goal to achieve a more dynamic business environment at the national and subnational levels. National documents mention progress achieved as mir- rored in DB improvements and use some DBIs as results indicators. The rapid feedback and simple nature of the DBIs has also provided backing for policy proposals and a means to keep reform momentum. DB has also served as a reform dialogue starter in public-private sector conversations and within branches of government. Globally known, DBIs are typically discussed by the national and local press and used by the government to showcase reforms internationally. Relevance of DBIs in Colombia is limited by the fact that the DB case study is not representative of the Colombian business sector reality The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix C both in terms of size (90 percent of businesses in Colombia are micro and small) and location (as Bogotá concentrates largest firms). » Doing things right: Is DB being used by clients and the Bank Group in ways to achieve the best effect for business environment reform? Paying taxes, trading across borders, and starting a business were the areas with the largest number of reforms, in some cases accompanied by score increases. The paying taxes score improved during the period as total tax contribution decreased by 14 percent and the number of payments were halved as a result of tax reforms implemented during the second half of the period. In the area of trading across borders, despite several administrative reforms, there was almost no change in the corresponding DB score, which is mostly affected by high transportation costs due to infrastructure gaps not affected by these types of administrative reforms. Finally, Colombia’s start- ing a business score continued to increase. However, after improving rapidly in DB2009 and DB2010, it continued to grow at a slower pace than in other economies, which led to a fall in its ranking. (Colombia is ranked 95th by DBI.) A key event was the creation, in 2008, of the “simplified corporation,” a new type of company operating under a law independent of the commercial code, which allowed for a reduction in transaction costs, time, and proce- dures to register a business. This “simplified” form is not captured by DB’s base case scenario that describes “a limited liability company” that “has five business owners.” 130 Two reforms were accompanied by a significant increase in Colombia’s getting credit and resolving insolvency scores and rankings. The most im- portant reform in the getting credit area, as measured by DB, was the new secured transactions law establishing a functional secured transactions system launched along with a centralized, notice-based collateral registry. As described in the DB2015 report, the law broadened the range of assets that can be used as collateral, allowed a general description of assets granted as collateral, and established clear priority rules inside bankruptcy for secured creditors. Its adoption resulted in a significant increase in Colombia’s get- ting credit score and ranking, which jumped from top 40 percent in DB2010– DB2014 to top 1–6 percent since DB2015. Regarding resolving insolvency, an amendment of the regulations governing insolvency proceedings to simplify the proceedings and reduce their time and cost also had a significant impact in terms of DB score (from 58 in DB2010 to 71 in DB2020) and ranking (from 32nd in 2010 to 12th in 2012). These improvements contrast with those of Colombia’s starting a business score after the large regulatory framework change, creating the SAS, which is a corporation type that offers greater flex- ibility than a standard LLC (Marechal et al. 2020). Lessons Colombia has successfully used DB to its advantage as a result of, in part, (i) long-term commitment and continuity, as Colombia has followed a regu- Independent Evaluation Group World Bank Group    131 lation simplification strategy throughout several administrations; (ii) in- stitutional capacity, since the government has a high-level understanding of the indicators and their scope and generally uses DBIs to inform specific policies; and (iii) strong political and technical support of DB-related re- forms. Yet there are some challenges for the indicators to remain relevant: (i) tracing of the impact of reforms beyond DBIs has been limited, as chang- es were not accompanied by an evaluation strategy to follow up on reform impact beyond what DB allows; (ii) inconsistencies and opacities in the DBI methodology has generated frustration among users, eroding some trust in the indicators, which could render them less relevant in the future; and (iii) inconsistencies with independent indicators and anomalous results have raised questions about the validity of some DB subindicators. As a result of reforms to getting credit, Colombia became one of the highest-ranking countries globally in this DB business area. This success, however, contrasts with some other access-to-credit indicators, based on which Colombia would be ranked below peer countries. Important reforms in other areas (for exam- ple, starting a business) have not translated into such significant score and ranking improvements. Democratic Republic of Congo » Doing the right things: Is DB doing the right things in terms of indicators and the country context? The Democratic Republic of Congo’s business climate remains a difficult one The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix C due to a wide range of factors, including administrative burdens, complexi- ty of taxes, and regulatory uncertainty. A significant business environment challenge is the gap between regulatory requirements (de jure) and real practices undertaken by firms on a regular basis (de facto). Businesses also face challenges due to poor infrastructure and a weak and corrupt bureaucra- cy. In addition, the Democratic Republic of Congo court system is often very slow to make decisions or follow the law, allowing numerous investment disputes to last for years. Property rights and contract enforceability pose se- vere limitations to investment, particularly for new investors trying to enter the market. Enterprise development in the country is heavily hampered by cumbersome and costly administrative procedures. In addition, a weak level of investor protection resulting from poor mechanisms enforcing contracts is a critical constraint to attracting private sector investments. The government’s motivation to engage in DB reforms was informed by goals to build the Democratic Republic of Congo’s brand image regionally and internationally, attract foreign direct investment, promote the domestic economy, enhance status, and improve the business environment. Based on stakeholder discussions, the key motivations for DB reforms include the de- sire to improve the country’s business environment in order to (i) build the Democratic Republic of Congo’s image regionally and internationally, espe- cially as the country emerged from decades of conflict; (ii) make the country attractive to potential foreign investors; and (iii) contribute to the devel- opment of domestic investors. The National Private Sector Development 132 Plan (2018–22) also confirmed the government’s commitment to fostering a sustainable and inclusive growth. Over the years, the government’s leadership on DB-related reforms has been evolving. From 2009 to 2015, reforms were led by the Steering Committee for Improvement of the Business and Investment Climate (CPCAI). From 2015, the Investment Promotion Agency (ANAPI) assumed leadership. In addition, President Félix Tshisekedi created a presidential unit to lead business reform and improve the Democratic Republic of Congo’s standing of 183rd out of 190 countries (DB2019). DBIs have been instrumental in identifying key constraints to the business environment and have influenced debates. DBIs have been used to influence business environment reforms primarily as a justification, discussion driv- er, and reputable source. Indeed, the DBIs played a key role in influencing discussions on business constraints, as highlighted by President Tshisekedi during a State of the Union address in 2019: “I will ensure that we improve the business climate by introducing an institution to monitor indicators.” » Doing things right: Is DB being used by clients and the Bank Group in ways to achieve the best effect for business environment reform? Despite enacting many reforms during the evaluation period, the Democratic Republic of Congo’s ranking on the ease of doing business (EoDB) indicator largely remained the same. The lackluster performance in EoDB (from 182nd Independent Evaluation Group World Bank Group    133 in 2010 to 183rd in 2020) resulted in part from negative reforms. As the country enacted positive reforms, it also recorded negative ones that slowed down its momentum. In the period between 2009 and 2019, along with the 31 positive reforms recognized by the DB report (DB2010 and DB2020), the country regrettably recorded eight negative reforms. Other explaining factors might be changes in DB methodology, which did not reward comple- mentary reforms (FY13–16 Country Assistance Strategy). Starting a business is the area in which the Democratic Republic of Congo has made the most progress. The country rose 100 places in this business area from being ranked 154th (DB2010) to 54th (DB2020), as a result of the large number of reforms recognized by DB. For context, informal micro, small, and medium enterprises dominate the country’s private sector and face significant barriers to growth and competitiveness. Over 90 percent of firms are small (one to nine employees), and nearly half of them have been on the market for less than five years. Yet firms six years and older con- tribute most to employment in the Democratic Republic of Congo (around 60 percent). Young firms account for over 35 percent of total employment. Reforms in the area of dealing with construction permits were focused on reducing the time and cost of processes and introducing the use of electron- ic systems to computerize processes. Reforms included improving building quality control and reducing the time it takes to obtain a building permit (DB2016). The country also made the process less expensive by halving the cost to obtain a building permit in DB2016, following similar reforms in The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix C DB2011 to reduce the cost of a building permit from 1 percent of the esti- mated construction cost to 0.6 percent and imposing a time limit for issuing building permits. Despite Bank Group and Democratic Republic of Congo at- tempts, there were limited reforms tracked in the area of enforcing contracts. Lessons Some relevant factors that explain the country’s performance were political and institutional instability, the consequential loss of reform champions within the government, and a public health crisis. National-level political instability led to two main consequences. One was high-level client disen- gagement from reform, and the other the loss of a business environment reforms supporting high-level leader. Meanwhile, important external factors included public health crises (Ebola and COVID-19). Together, those led to the discontinuation of reforms, mainly impacting the enforcing contracts business area. Factors of success that contributed to the implementation of reforms in the Democratic Republic of Congo, on the other hand, include (i) institutional ownership, whereby ANAPI is directly responsible for the reform process; (ii) the presence of reform champions at all levels; and (iii) constructive public-private dialogue. 134 India » Doing the right things: Is DB doing the right things in terms of indicators and the country context? India is a lower-middle-income country with increasing relevance and per- sistent development challenges. It is the eighth-largest country in the world and the second most populous, with 1.3 million inhabitants and a per capita gross national income of $2,120 as of 2019. For the period of this evaluation, the government of India promoted fiscally conservative policies, registering steady economic growth and resilience, including through the 2009 global economic crisis. While the country developed space technology, high-quality information technology services, and pharmaceutical industries, most of its population still work on small subsistence farms and in informal, basic, low- skill workshops. Other important development challenges are low access to finance and to land, and high corruption levels. The DB indicators have spurred reforms in India, especially since 2014. The current government has focused its efforts on improving India’s global rank in DB and has set a specific target of being among the top 50 countries in the world. To achieve that, it asked a department in the Ministry of Com- merce and Industry—the Department of Industrial Policy and Promotion (DIPP)—to liaise and work with the Bank Group. With its specific focus, DIPP gained a deeper appreciation of the value of DB as a diagnostic and a Independent Evaluation Group World Bank Group    135 benchmarking tool. The World Bank subnational DB of 2009 and the World Bank enterprise survey pointed to big differences among the states in India. In 2015, DIPP rolled out the Business Reforms Action Plan for state govern- ments and union territories in India to improve the EoDB and the ease of regulatory compliance for businesses across the entire country. The program for reforms, in partnership with state governments, aimed to make it easier, simpler, and quicker for businesses to operate, and inspired by DB, the states and their reforms are rated annually and the states are ranked. Both the World Bank and IFC carried out advisory programs to help the central and state governments improve their EoDB. Despite recent controversies about DB and its methodology, in India DB has a reputation for being objective and specific. However, as in other case coun- tries, critics have pointed to the limitations of the DB indicators in covering key constraints facing businesses in the field, and the limitations of DB’s approach of representing conditions in only two cities (as it does elsewhere) given such diverse conditions nationwide. » Doing things right: Is DB being used by clients and the Bank Group in ways to achieve the best effect for business environment reform? While the rapid rise of India in rankings—from 142nd in 2014 to 63rd in 2020—is welcomed, some experts expressed concern that, while some of the reforms implemented met the specifications of the DB indicators, the actual progress in the field may not have matched the movement in the indicators. The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix C India implemented 59 reforms, many of which were linked to national plans for increased digitization of the economy. There was a preponderance of reforms with easy procedural simplification in starting a business, paying taxes, trading across borders, and dealing with construction permits, which reduced the time and the costs involved. There were fewer reforms success- fully implemented in the more difficult areas involving legal processes, such as in resolving insolvency, registering property, and employing workers. However, one complex reform area highlighted in interviews was resolving insolvency, where a new legal framework was developed and introduced. Indicative of the long-term nature of these challenges, despite major reforms to improve India’s insolvency framework, business associations reported the persistence of slow, cumbersome, and inefficient resolution in court. DB also helped the government develop specific targets and the means to monitor them. The long-term relationships between Bank Group staff and DIPP offi- cials have been a positive factor for the reforms. For many business stakeholders consulted in India, the DB reforms were ade- quate but not very helpful. They welcome that DB has motivated relevant re- forms but, in their opinion, DB does not cover many issues they perceived to be important to them, such as poor infrastructure, problems with land, labor, and governance. Foreign investors pointed out that they do not base their investment decisions solely on the DB indicators as a gauge of the business environment. They point to the lack of coverage in DB of critical themes in- cluding a level playing field, the role of the state in business and the support 136 for SOEs, and the negative lists of areas where foreign participation is not permitted. They indicated that their focus was on the predictability of the legal process in the country, repatriation of their earnings, civil unrest and macro-economic factors including the volatility and trend of exchange rates. Lessons Five main lessons can be derived from the experience in the country. One, specificity offered by the DB (sub)indicators was critical to drive medium- and long-term reform, as they would guide policy making. Two, even simple reforms may require interministerial coordination and high-level oversight, as many of the reforms had to be implemented by two different state and municipal governments and across different ministries. Three, simply imple- menting reform is not sufficient, as monitoring, feedback, and public-private dialogue would enhance adequate rollout, design, and awareness. Four, am- bitious reform programs required flexibility in design and implementation, as they often involved separate ministries, departments, and even interest groups. Five, competition is good. Ranking states enhanced incentives for reform. Competition between the two cities captured in DB helped drive the reform process. Indonesia Independent Evaluation Group World Bank Group    137 » Doing the right things: Is DB doing the right things in terms of indicators and the country context? Indonesia has consolidated as an upper-middle-income economy with a fast-growing private sector and regional influence, but also with regional and income inequalities. Since 2000, boosted by a surge in the price of key commodities, Indonesia has emerged as a vibrant and stable democracy. In 2019, Indonesia had a total population of 270 million, with a life expectancy of 71.5 years. The country’s GDP has increased by more than 400 percent since 2000, with a subsequent decline in the poverty rate from 24 percent in 1999 to 10 percent in 2018. However, the rapid economic growth has led to increased inequality, with large geographical disparities and a Gini index of 38. The distribution of inequality remains highly concentrated. There are more than 300 ethnic groups, with the Javanese being the largest group (42 percent), followed by the Sundanese (15 percent). In this context, Indonesia’s business environment is marked by a combi- nation of micro, small, and medium enterprises and large SOEs. Both types suffer from low productivity and exhibit limited integration into regional and global value chains. The SOE sector plays a significant role in the econ- omy, and SOEs’ interests greatly influence economic policy. SOEs receive subsidies and operate as monopolists or dominant players in key sectors. Furthermore, Indonesia has some of the tightest restrictions on foreign direct investment among middle-income countries surveyed by the OECD, which inhibit market entry, diminish commercial performance, and increase The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix C prices. Indonesia’s weak competition framework prevents authorities from effectively discouraging anticompetitive behavior. DB has strongly influenced national development planning. In 2017, Presi- dent Joko Widodo reaffirmed his EoDB 40th rank goal and instructed min- istries to draft detailed plans to achieve it. As a response, in 2019, the head of the Ministry of National Development Planning (Bappenas) announced a gradual transition into e-bureaucracy, reflecting particularly on the licensing process for the investment sector. Furthermore, in recent years Indonesia has introduced an online platform for business licensing, replacing hard cop- ies with electronic certificates. The country also introduced online systems to file and pay major taxes, process export customs declarations, and manage cases for judges supporting contract enforcement. Recently, the Congress approved the Omnibus Law on job creation. Primarily, it streamlines the complex regulatory environment in 11 critical areas, including labor law, capital investment, business licensing, corporate tax, and land acquisition. The influence of DB in the Bank Group portfolio in Indonesia is unique. It is highly concentrated in volume by a sequence of development policy loans (DPLs) supporting an agreed reform program, clearly defined in both the CPF and the government’s planning documents. Additionally, DPLs are com- plemented by significant support in a high number of advisory services and analytics and analytical work projects. DBIs have been useful for identifying business climate constraints related to several procedures, time, and costs. From the 1960s to the late ’80s, interest groups in Indonesia pushed the 138 reform agenda into specific political and economic directions, many of them highly ideologically weighted. Since its inception, DB has provided an inter- national benchmark that has influenced the main political stakeholders to support a more pragmatic reform program, smoothing the policy dialogue on the country’s priorities and the best way to address them. The most recent example was the mentioned Omnibus Law, supported by a diverse majority of political forces. However, Indonesia has focused its business-oriented reforms on those mea- sured by DBIs, marginalizing crucial and more comprehensive development components such as security, macroeconomic stability, reduced corruption, labor skills, quality of institutions and infrastructure, economic competition, investment, and trade policies (other than import/export time and cost). Although DBIs appear to have helped pave the way for a solid path of reforms, this has been a risky approach, enhancing a biased view of a successful reform- ist case and diverting attention away from anticompetitive public rules and practices which should have been addressed. The limited DB scope has failed to grasp the whole performance of the country’s business environment. » Doing things right: Is DB being used by clients and the Bank Group in ways to achieve the best effect for business environment reform? Although DB has helped as an entry point with government, its rankings might be overshadowing business areas in need of policy reforms. For in- Independent Evaluation Group World Bank Group    139 stance, DBIs have raised concerns in their accuracy of capturing the coun- try’s situation regarding insolvency issues. In the last five years, Indonesia’s score in resolving insolvency has remained almost the same (67.7 on aver- age) and well ranked (38th in 2020). While achievements in this regard can be identified (for example, the 2004 Bankruptcy Law), in practice improve- ments in the legal framework and law implementation are still pending. The government’s priority of enhancing investment attraction has become in- creasingly translated into improving Indonesia’s position in the DB ranking, which has proved to be a misleading path for identifying priorities in reform areas and improving the business environment. This situation is also confirmed by the divergence of results on DB in In- donesia presented in other indexes (for example, the Japan Bank for Inter- national Cooperation survey). Contributions of DB-related reforms to the business environment might be difficult to associate from a causal per- spective. However, normative changes followed by movement in indicators might capture a part of the DB expected effects. Empirical analysis suggests that reformed areas have directly improved DBIs. Getting credit, starting a business, and paying taxes are among the indicators reflecting a major im- provement from 2015 to 2020. These indicators alone amount to 52 percent of DB-related reforms since 2010. However, it should be noted that trading across borders reforms added 10 percent of total reforms for the same period, The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix C but only registered a 4.7-point improvement in the last five years. Lessons In Indonesia, DB has been highly successful as a discussion driver between the Bank Group and policy makers, and between the government and the main political stakeholders in the country. The controversy on the practical results of reforms influenced by DB points out a usage problem that goes beyond the DB scope as an instrument. Empirical evidence indicates that in the absence of DB, other indicators and rankings, such as the World Economic Forum’s Global Competitiveness Report (which is considered to be methodologically less rigorous), would have been used to guide and prioritize specific topics with- in the country’s reform agenda. Most of the stakeholders interviewed by the Independent Evaluation Group (IEG) agree that considering that the DB scope is limited, other Bank Group products (that currently have a lower profile compared to the DB) should complement it to capture a more comprehensive analysis of the business environment. Instruments such as the Country Policy and Institutional Assessment and the enterprise surveys are generally reliable and can correlate with DBIs, strengthening the analysis and providing a better understanding for Bank Group staff and clients. 140 Jordan » Doing the right things: Is DB doing the right things in terms of indicators and the country context? Jordan is an emerging market economy with slowing growth levels, increas- ing population, and rising public debt, leading to worsening poverty and un- employment. Indeed, unemployment has risen to almost 25 percent in 2020 and affects youth, university degree holders, and women much more nega- tively, further contributing to inequalities. Jordan’s economic resource base centers on phosphates, potash, and fertilizer derivatives; tourism; overseas remittances; and foreign aid. These are its principal sources of hard cur- rency earnings. Lacking alternative energy and water supplies, Jordan relies on natural gas for 93 percent of its domestic energy needs and suffers from regional instability. Nevertheless, it has benefited from international aid as the country has become a central element of stability in the region, ensuring peace on the borders it shares with its neighboring countries. Jordan also has multiple industrial zones producing goods in the textile, aerospace, defense, information and communication technology, pharmaceutical, and cosmetic sectors. Jordan has an emerging knowledge economy. Jordan’s ranking in EoDB remained relatively stable during the whole de- cade, oscillating between 105th and 126th position. Similarly, the EoDB score slowly and gradually increased from 56.3 in 2010, when the country’s Independent Evaluation Group World Bank Group    141 ranking was 108th, to 61.3 in 2019, with a ranking of 102nd. In the 2020 EoDB, Jordan marked a record performance by going up 29 ranks (from 102nd in 2019 to 75th in 2020) and was recognized as one of the 10 most improved economies. This progress has mainly been led by improvements in the getting credit, paying taxes, and resolving insolvency indicators. The getting credit score increased from 35 in 2019 to 95 in 2020 (moving Jordan’s country rank for getting credit from 134th to 4th), mainly thanks to the im- plementation of the Movable Collateral Law, which improved the legal rights index of the indicator from 0 to 11, and the Insolvency Law, which improved the credit information index. Overall, international rankings, reform plans, and executive directives are often detached from the actual needs of the private sector. (EBRD) This indi- cates that though the country needs to reform its legal framework and create incentives to stimulate Jordan’s private lending environment, the reforms implemented or proposed through the Bank Group’s operations do not tackle the correct issues. This also means that, though the projects target areas that need to be reformed, the reforms proposed mainly focused on increasing the country’s ranking on the DBIs rather than being in line with the private sector’s needs. » Doing things right: Is DB being used by clients and the Bank Group in ways to achieve the best effect for business environment reform? Efforts to improve starting a business fell short, in spite of support from a The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix C 2009 DPL and two subsequent ones. The 2012 and 2014 DPLs did not fully reach their starting a business–related goal of cutting the time and required licenses. It is unclear which of the implemented reforms would impact this target. On the other hand, reforms related to paying taxes and getting credit led to good results. Jordanian tax authorities gradually adopted an electron- ic system for filing and paying taxes, making its use mandatory from 2018. This has significantly improved its annual hours per tax payment subindica- tor. Further, through IFC’s 2020 advisory operation, Jordan will also adopt a risk-based audit by creating an engine based on risk management. This will contribute to improving the country’s postfiling subindicator. In addition, the government started keeping a unified registry for collateral that is notice based. This simplifies the process and improves the quality of registration by minimizing errors due to fewer documentation requirements. The laws also allow the country to adopt a functional approach to secured transac- tions, according to which rights of movable assets that secure the payment or performance of an obligation become functional equivalents to traditional types of securities. Regarding the credit information index, the 2010 Credit Information Law and the 2016 private Credit Bureau introduced all the good practices laid out on the DB website, such as reporting both good and bad in- formation and eliminating minimum loan thresholds. Since 2018 the bureau also started receiving data from retailers to establish a good credit history for those who do not have previous bank loans or credit cards and providing credit scores as an additional service offered. 142 Jordan made good reforms in getting credit and starting a business, despite the challenging context. Jordan adopted several different good practices through the 2018 Movable Property Law and Insolvency Law, allowing a general description of collateral, maintaining a unified registry, and pro- tecting secured creditors’ rights. To improve conditions to start a business, it reduced or eliminated the paid-in minimum capital requirement; created or improved one-stop shops; and simplified registration processes, using electronic services. Lessons During the whole evaluation period, Jordan was subject to numerous exog- enous shocks that delayed implementation of DB-related reforms. An eco- nomic slowdown, caused by the global recession and regional crisis, shifted near-term priorities and moved government focus to other areas, especially management of public investments, taking attention away from DB-related issues. As well, cabinet reshuffling increased the difficulty of quickly adopt- ing reform. On the other hand, committed, dynamic, and forward-looking ministers in relevant ministries were crucial in working with the Parliament to push for the implementation of reforms related to getting credit. Above all, a big success factor was the perseverance shown by both the Bank Group team and the government in pushing for these reforms in a time difficult for Jordan, reforms that were eventually implemented. Independent Evaluation Group World Bank Group    143 Failures identified in projects related to starting a business include timing of analysis and design. Much of the background analysis on which the 2009 DPL was based was carried out years before the global financial crisis and eco- nomic slowdown. This applied to both starting a business and paying taxes reforms. Failures in subsequent DPLs of 2012 and 2014 could derive from the design of the projects: either choice of a target that did not absorb the effects of the implemented reforms, or choice of reforms that did not focus on decreasing the time needed to start a business. Morocco » Doing the right things: Is DB doing the right things in terms of indicators and the country context? Morocco is a lower-middle-income North African nation with the sec- ond-largest economy in the Arab world. Traditionally rural and agricultural, it has transitioned to a majority urban economy. Over the past 20 years, it has achieved relatively strong economic and social progress thanks to re- forms that helped to stabilize its macroeconomic framework. These reforms include phasing out energy and food subsidies, improving fiscal and financial policy frameworks, and promoting economic diversification and competi- The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix C tiveness. Morocco also achieved some success in promoting high-value-add- ed manufacturing sectors (for example, pharmaceuticals). The country’s business environment remains bureaucratic, subject to ar- bitrary decision-making, and lacking in competition. Heavy and opaque administrative formalities lead firms in enterprise surveys to identify cor- ruption, tax administration, and informal competition as leading constraints. The surveys indicate substantial constraints to SMEs; for example, they spend two to three times as much of their management’s time dealing with government regulations as do large firms and are more likely to find infor- mal competition. In the broader economy, oligopoly and lack of competition are important constraints, making it difficult for new firms to enter sectors that elsewhere are typically characterized by low market concentration, such as manufacturing. Finally, weak governance is a persistent constraint. After having sharply criticized DB until around 2007, the country enthusi- astically embraced DB-informed reforms and from 2010 to 2020 is credited with an impressive 33 positive reforms as captured in the indicators. Observ- ers found DB rankings highly motivating to their counterparts, to the extent that it was prioritized over more tangible economic outcomes. The govern- ment established an objective to be among the top 50 countries for EoDB by 2021. While reforms covered many areas, they prioritized starting a business and paying taxes. Morocco advanced in its EoDB ranking from 128th in DB2010 to 53rd in DB2020. 144 The World Bank Group has provided technical and financial support for Mo- rocco’s DB-related reforms through eight identified projects over the evalua- tion period. The DB agenda was also treated in broader country strategy and economic documents. Several activities were both broader and deeper than the DB agenda. The FY19–24 CPF echoes the findings of the SCD and the Country Private Sector Diagnostic (CPSD), emphasizing areas mostly beyond the DB agen- da: corruption, lack of a level playing field, competition from the informal sector, low workforce education levels, and difficulties in accessing financ- ing. It notes the government’s 2021 EoDB goal but establishes a strategy well outside of this. Under the CPF, DB forms a clearly acknowledged but small part of an ambitious agenda. The CPF speaks of a new generation of reform: “A combination of lending, advisory and analytical engagements across IFC and World Bank will seek to facilitate second generation business regulatory reforms to reduce further the cost of doing business, support the digitization of business services, enable fairer competition, support contestability and strengthen implementation capacity of policy making.” A theme is one of diminishing relevance of the DBIs over time. Through much of the evaluation period, the National Business Environment Com- mittee (CNEA) saw great benefit to the DBIs as a focal point of reform and saw improving Morocco’s standing as a “bottom line” for reforms. It saw the DBIs’ improvement as a clear way to communicate reform success to for- Independent Evaluation Group World Bank Group    145 eign investors and donors. Thus, DB was initially quite useful in initiating dialogue, opening areas of reform, providing agreed metrics and monitoring standards, and coordinating efforts through the leadership of CNEA. It pro- vided entrées in some cases to broader reforms in areas covered by DB. Certain limitations both to DB and to CNEA’s policy motivation became of greater concern. First, some Bank staff raise a question about the contradic- tion between DB’s focus on the experience of a typical SME described in its cases and the government’s effort to attract foreign investors, who generally did not operate under the regimes described. As low-hanging fruit was pro- gressively addressed, the remaining DB agenda became one of longer-term or less politically palatable actions, including work on the legal and judiciary system. In addition, as the business-government interface became increas- ingly digitized, the static measures of DB became less relevant to the under- lying procedures they were trying to capture. Finally, and most importantly, there was the recurring critique that the DB agenda was mostly orthogonal to the most pressing constraints on domestic businesses (CPF, CPSD, SCD, World Bank staff interviewed). Doing things right: Is DB being used by clients and the Bank Group in ways to achieve the best effect for business environment reform? Morocco’s reforms yielded strong advances in its DB ranking (in 2020, 53rd in EoDB), and DB is understood to have uniquely influenced the govern- ment’s reform agenda, easing procedural compliance in multiple areas. DB The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix C motivated a great deal of reform activity, enhanced government coordina- tion, and laid the groundwork for more systemic monitoring of reforms. By DB2020, Morocco was ranked 16th in getting construction permits and 24th in paying taxes. Observers pointed to the influence of DB in engaging re- forms, noting “there is nothing similar” among other indicators or diagnos- tics. At the same time, experts observe that it remains difficult for SMEs to enter many markets, access key inputs, and contend with bureaucratic and legal procedures. Lessons Overall, Morocco has used DB to its advantage as a result, in part, of internal and external factors. These included the following: Internal: (i) Concrete targets. In general, the DBIs are seen to give specif- ic targets that are useful to government in setting targets and monitor- ing progress. The measured progress of DB at the national level has now spawned support for subnational reforms extending to the Marrakesh region and beyond. And the recent pilot extension of DB to contracting with gov- ernment has facilitated engagement on such issues as government payment arrears to private businesses. Yet the rigidity of targets could be limiting in what reforms were credited or even supported by monitoring. (ii) Mobilizing specialized expertise. The central indicators-based reform team developed experts who could be mobilized to assist various national reform efforts. In some cases, specialized expertise was mobilized to help develop digital/ 146 information technology systems to support reforms. (iii) Addressing appro- priate levels of government. Given that DB measures a main city or cities, the appropriate forum for effecting change for several indicators was in fact regional or municipal government. (iv) Continuity. The Bank Group’s contin- uous presence and dialogue across various instruments and periods seems to have been a positive factor. Long-term relationships, trust, and communi- cation developed between key staff and their counterparts. (v) Coordination. IFC and World Bank staff working on the business environment were part of joint units and, as such, were able to mobilize complementary instruments (for example, development policy operations, analytics, advisory support) within a common strategic framework. The recent split of IFC advisory ser- vices staff from joint practices has somewhat weakened this ability. External: (i) Client capacity. The counterpart developed a sophisticated understanding of how to move Morocco’s DB rankings and how to use the indicators to motivate, coordinate, and monitor reforms. CNEA was created with encouragement from the Bank Group, which also advocated for pub- lic-private consultation. CNEA became skilled at incorporating public-pri- vate dialogue into the reform process (broadening ownership), constructing action plans with defined and assigned tasks and deliverables, overseeing implementation, and evaluating progress. Public-private dialogue in itself was credited as a source of reform success, bringing greater consensus and stronger ownership of reforms. (ii) Client commitment. The strong motivation Independent Evaluation Group World Bank Group    147 of the counterpart to improve DB rankings gave substantial impetus to relat- ed reforms. It came from “the highest level” of government, and with it came accountability, coordination, and the ability to overcome many obstacles to implementation, mobilizing actors and speeding adoption of new laws and regulations. At the same time, limitations to counterpart commitment meant some difficult issues were not addressed. (iii) Cultural change. One benefit of DB, according to a key staffer, is that “DB is contributing to a new way of doing. A DB-like dynamic could be extended to other sectors. You could have units similar to CNEA with clear indicators and objectives [although this is] not common.” Said another staffer, “There is not in Morocco a culture of close monitoring and evaluation in anything. Other Bank Group products don’t have the same influence and power in convincing government, in cre- ating a special delivery unit.” Russian Federation » Doing the right things: Is DB doing the right things in terms of indicators and the country context? The Russian Federation is an upper-middle-income non-OECD country char- acterized by global economic relevance and governance-related challenges. The country has the largest surface area in the world, holds the 11th-biggest GDP, and has the 9th-biggest population, with outstanding natural resourc- es that stem from its vast geography. It is a major emerging economy with significant international influence and a diversity also characterized by an uneven territorial development. Russia is also the third-biggest oil and gas The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix C producer and exports other commodities such as coal and wheat, which generates income while also characterizing a resource-based economy. Meanwhile, it is home to deep-rooted oligarchs who maintain close ties with the government. Indeed, some of the country’s main vulnerabilities relate to governance, and recent conflicts. These were aggravated by 2014’s conflicts over Crimea and the consequential set of international sanctions imposed by major developed economies, which also affects the economy at other levels such as foreign investment. Russia counted on both the national and Bank Group plans aligned with the improvement regulatory environment. The Bank Group’s Systematic Country Diagnosis (2012–16) outlined priorities including streamlining regulatory requirements to enhance conditions for entrepreneurship, competition, and SME growth. That resonated with the presidency’s target in the 2012 “May decrees” of improving the Russian DB rankings. Wide support from higher government officials was complemented by the role of the Ministry of Economic Development in leading the process along with a nongovernmental organization, the Agency for Strategic Initiatives (ASI). While led by the presidency, they coordinated internal and external actors as well as public-private dialogue, including experts and the Bank Group. Until recently, the framework was organized around 12 yearly roadmaps (7 related to DBIs), tools that carried mutual agree- ment of parts involved and that worked for transparency, implementa- tion, and follow-up measures. 148 World Bank support was mainly delivered through a flexible programmatic assistance package comprising technical assistance, RAS, and just-in-time (or ad hoc) interventions. Major support was provided by World Bank local and central offices, often focused on the electricity and construction permits indicators. The support given was largely based on a pipeline of assistances that provided the country with reform memos guiding reforms, regular sup- port to clarify methodologies (in some periods, daily calls with ASI), intro- duction to best performers with contexts that were analogous to the Russian, and attendance at roadmap-related meetings. Areas supported by the World Bank were mainly DB related but were not limited to them. The World Bank has identified the risk of over-focusing on DB and developed mitigation efforts, which include clarifying that the mere improvement of the rankings would not lead to business environment improvements, and developing sup- port on other fronts such as SME development and competition policies. » Doing things right: Is DB being used by clients and the Bank Group in ways to achieve the best effect for business environment reform? The country implemented a striking total of 53 reforms, structured around a highly hierarchical, organized, and inclusive framework. Russian rank- ings and government effectiveness increased strikingly, but despite such improvements, external trust in the business climate is still affected by governance issues and sanctions. The Russian EoDB had a 77 percentage point increase in the period, an impressive result of its coordinated efforts Independent Evaluation Group World Bank Group    149 in several areas, led by getting electricity and dealing with construction permits. Similarly, its United Nations Development Programme government effectiveness rating also increased 46 percent in the period. However, Russia still faces internal challenges related to corruption and governance. Exter- nally, its GDP and investment levels face an exogenous force that DB cannot influence: the sanctions imposed by the United States and European Union in 2014 following the armed conflict over Crimea. The Russian government has also developed its own adapted subnational yearly DB report and awards. During the evaluation period, the executive branch of the federal government has implemented a strategy of coordina- tion and incentives’ alignment of subnational leaders to improve the busi- ness climate. Along with the ASI it established a national ranking similar to DB but adapted to the Russian subnational context. With yearly recognition of best state performers, it gained a lot of attention from competing gover- nors and the public. It can be said that the business environment reforms in Russia ended up outgrowing the DB, adapting themselves to local develop- ment challenges and to mitigating regional inequalities. Lessons DB worked as a tool to inspire change, open dialogue, identify constraints, and monitor progress. Success factors were the complete buy-in from the presidency (including setting targets and plans by decree) in coordination with lower levels of national and regional governments; high-level frequent The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix C monitoring of midlevel implementation; the agenda overlap between DB and the government’s plans; the ASI’s convening of the public-private dialogue; the engagement of educated civil servants; the inclusion of other, non-DB dimensions in business environment reforms; the capacity-building efforts with just-in-time (or ad hoc) support (including methodology clarifications and simulation of reforms’ effects over scores), peer learning, and business reform memorandums; the pipeline of flexible and responsive RAS projects; and the virtuous competition that was established. The positive effects of peer-to-peer learning with government agents from well-performing coun- tries identified by the Bank Group were emphasized by government mem- bers of different DB areas as a vital success factor allowing the agencies to find advice and inspiration from other reformers. World Bank local officials complained of a lack of resources for support to reforms, which led to an executive decision for a team focus on dealing with construction permits and getting electricity. Stakeholders repeatedly confirmed not perceiving conflict of interest in the DB measurement and operation. Rwanda » Doing the right things: Is DB doing the right things in terms of indicators and the country context? Some of Rwanda’s structural characteristics constitute major constraints in the path toward development. The country is landlocked, with the highest 150 population density in mainland Africa. The International Monetary Fund expects Rwanda will reach a gross national income per capita of US$821 in 2021, ahead of 12 of the 48 countries in Sub-Saharan Africa, but it is still a low-income country. While urbanization efforts in the past years have been particularly directed toward Kigali, its capital city, around 80 percent of its population live in rural areas. Reducing dependence on foreign funding represents a major challenge for the country. In the aftermath of the genocide that took place in 1994, the govern- ment focused its efforts on rebuilding the country, thus making continuous public investment the major driver of economic growth. At the same time, it became a highly favored recipient of foreign development assistance, which has been declining in the last years; consequently, private investment and do- mestic savings must increase as well as tax revenue. In addition, creating pro- ductive jobs constitutes Rwanda’s main development challenge (CPSD 2019). High population growth highlights the need for creating productive jobs to prevent higher unemployment and poverty rates. To address these challenges, the government needs to promote the business environment. The government’s strategic development agenda has been fundamental for Rwanda’s progress in the past two decades. Economic prosperity has been the cornerstone of the strategy, as it has contributed to legitimizing the po- litical system and gaining international recognition. During the evaluation period, different plans and strategies were adopted, generally embracing a Independent Evaluation Group World Bank Group    151 broad development agenda, such as the United Nation’s Millennium Devel- opment Goals and Sustainable Development Goals. All of them recognize the importance of the private sector in leading the country’s economic growth. The influence of the DB report is clear in some national plans and strategies. Official documents (for example, the Economic Development and Poverty Reduction Strategy [EDPRS] and Vision 2050) targeted reducing the average number of days to deal with licenses, strengthening “Rwanda’s Doing Busi- ness Brand,” and getting ranked in 10th place by DB2035 and maintaining the status afterward. The DB report has been used as a tool for prioritizing reforms, although its influence has decreased in recent years. Government officials use the DB report as a roadmap to recognize pressing issues and corresponding good practices, including peer countries’ benchmarking. Nevertheless, they also see the scope of reform guidance broadened beyond the DB report in recent years. Other sources of information are the National Institute of Statistics Rwanda, the Global Competitiveness Index (World Economic Forum), and the Financial Freedom Index (The Heritage Foundation). Country strategies have been aligned with national plans, with a strong fo- cus on improving the business environment. For instance, the Bank Group’s Country Assistance Strategy for Rwanda for FY09–12 was framed around the EDPRS strategic flagships and had a strong emphasis on supporting invest- ment climate reforms as measured by the DB report. The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix C In the IEG-identified, DB-relevant portfolio, the Bank Group financed six projects during 2010–20, totaling US$171.9 million in commitments. Most projects (five) were lending operations, while one was an advisory service provided by IFC. Most of the Bank Group’s support was directed toward get- ting electricity (four projects), paying taxes (two projects), and trading across borders (two projects). » Doing things right: Is DB being used by clients and the Bank Group in ways to achieve the best effect for business environment reform? DB reports recorded 51 major reforms that made it easier to do business in the country. Rwanda was among the top reformers during the evaluation pe- riod. Reforms were focused on getting credit (14 percent), starting a business (14 percent), and dealing with construction permits (12 percent), while em- ploying workers was the area with the fewest reforms (2 percent). The most common interventions addressed improving business laws and regulations (35 percent), supporting the use of electronic systems or automation (19 per- cent), and the reengineering of processes (19 percent). A positive relationship between DB-related reforms and immediate out- comes is clear, but this does not hold for higher-level outcomes. DB reforms were effective in enhancing DBIs, score and ranking. Rwanda ranked 38th in DB2020, jumping 29 positions since DB2010. The country is now the sec- ond-easiest place to do business in Africa behind Mauritius. Although it is unquestionable that progress has been made in terms of domestic invest- 152 ment and savings, credit to the private sector, foreign direct investment, tax revenue, and exports, the 2019 CPSD points out that some key cross-cutting obstacles (related or not to DB) have not been overcome yet. Persistent struc- tural constraints are low skill levels, limited access to and high costs of energy, high costs of transportation and information and communication technology, and restricted access to land. In addition, income per capita has not grown at the same pace as GDP, and some indicators suggest that the benefits of eco- nomic growth are still concentrated in a few segments of the population. Lessons While political will and strong coordination between public institutions were the most important factors behind reforms’ success, limited state capacity was a challenge. Government commitment was constant during the evalua- tion period, which helped to direct resources toward DB reforms. At the same time, the Rwanda Development Board played a fundamental role in coordi- nating such reforms between the executive, legislative, and judiciary powers, and with development partners as well. Yet Rwanda needs to improve the capacity of public agencies. According to the 2019 SCD, low levels of human capital in the country are reflected in an insufficiently trained bureaucracy, which negatively affects projects’ implementation. Likewise, the govern- ment’s top-down decision-making approach could be backfiring as it hinders innovation among public servants. Independent Evaluation Group World Bank Group    153 Addendum on Doing Business Reform Memorandum Analysis IEG examined the most recent available DB reform memorandums of the countries selected for case studies; hence, this “sample” is not necessari- ly representative of all the DB reform memorandums published in the last decade. However, aligning our selection with the case studies enhanced IEG’s learning potential. The quality of the memorandums was reviewed using primarily qualitative criteria—for example, analytical frameworks used to make recommendations, adaptation of the recommendations to the national conditions, coverage depth beyond DBIs, best practices cited, and the speci- ficity of the factors mentioned as shortcomings faced by countries. The memorandums have evolved over the last decade and display a more structured and standard format. The standardized presentation presents a trade-off. On one hand, the information is presented more clearly, allowing it to reach a broader number of policy makers and enhancing comparability among countries. On the other hand, that structure seems to encourage a cookie-cutter approach, which poses a risk to the recommendations’ depth and appropriateness. Only 3 of the 10 DB reform memorandums reviewed include a systematic use of non-DB analytical frameworks to support DB-oriented reforms and law regulations.1 These frameworks were mostly used as justification and as complementary evidence to support recommendations.2 However, the mem- The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix C orandums that included non-DB analytical frameworks were able to provide recommendations strengthened by a broader set of information. There is a high variance in the adaptation of DB reform memorandums to country conditions. Within the small sample, large countries, such as Chi- na, Russia, and India, benefited from a more tailored framework and a more comprehensive approach than smaller countries, such as Jordan, Morocco, and the Democratic Republic of Congo. Another aspect of tailoring is the use of best-practice examples in various sections of the DB reform memorandums. Some examples offered seem not to represent the level of administrative ca- pacity of the client. Examples include the case of the electronic identification in Estonia; the experience of Ontario, Canada, in developing a risk classifica- tion; and the “FastTrack” platform in Portugal that allows users to select a pre- approved name from the registry’s website. While the repetition of these cases is not necessarily incorrect, there is no clear indication of the way in which very diverse countries can benefit from these experiences or whether there is a particular logic in the repeated use of the same illustrations. Moreover, there is an evident discrepancy in the approach used by DB reform memorandums across different indicators. For some indicators, such as get- ting credit, the memo offers a broad set of similar recommendations for all countries. By contrast, other indicator recommendations, such as protecting minority investors, are adapted entirely to the country’s context. 154 In sum, DB reform memorandums present many useful and practical recom- mendations and tools to help improve DBI scores. However, many of these recommendations seem isolated and not linked to a deeper framework that would fully capture the challenges in each country’s overall business envi- ronment and within each DB area of focus. References China. 2006. 11th Five Year Plan (2005–2010) for National Economic and Social Development. https://policy.asiapacificenergy.org/sites/default/files/11th%20 Five-Year%20Plan%20%282006-2010%29%20for%20National%20Economic%20 and%20Social%20Development%20%28EN%29.pdf. Marechal et al. 2020. UNDP (United Nations Development Programme) Afghanistan. 2021. Achieving Afghani- stan’s Long-Term Goals amid Short-Term Adversities. Kabul: UNDP Afghanistan. Independent Evaluation Group World Bank Group    155 1  The three reform memorandums are for China, Russia, and India. 2  Mainly the World Bank enterprise survey, Logistics Performance Index and Global Com- petitiveness Index, Country Partnership Strategy, International Monetary Fund reports and working papers, Organisation for Economic Co-operation and Development Trade Facilitation Indicators, research papers. The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix C 156 Appendix D. Indicator Area Deep Dives Deep dives were developed regarding five specific Doing Business (DB) indicators: dealing with construction permits, getting credit, paying tax- es, starting a business, and trading across borders. They allow a careful probing of evidence on each business area and whether and how the indi- cators or reforms based on those indicators have been shaped by evidence and experience. Each deep dive explores indicator features, construct and content validity, the relevant portfolio’s design and effectiveness character- istics, and the role of other stakeholders at the country or global level. Each deep dive draws from a review of relevant findings from the portfolio review, the two literature reviews (desk and structured), the 10 country case studies, and interviews with subject matter experts and practitioners. Following the templates used for the evaluation deep dives, each background note is sum- marized in this appendix addressing, whenever possible, the following: » Doing the right things: Is DB doing the right things in terms of specific busi- ness areas? » Indicators’ features and content validity Independent Evaluation Group World Bank Group    157 » Main types of country reforms and World Bank Group interventions » Uses in drawing attention to reform, identifying and prioritizing reforms, guiding design, and monitoring and evaluation (M&E) » Doing things right: Is DB being used by governments and the World Bank Group in ways to achieve the best effect for business environment reforms? » Effectiveness » Factors of success and failure » Internal » External Cross-cutting findings from the case studies and deep dives are summarized in table D.1 at the end of this appendix. Dealing with Construction Permits The DB dealing with construction permits (DWCP) indicator measures the procedures, time, and cost to build a warehouse—including obtaining the necessary licenses and permits, submitting all required notifications, re- questing and receiving all the required inspections, and obtaining utility connections. Additionally, it gathers data on quality control based on six indexes. The methodology is based on a highly stylized case study (a ware- The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix D house construction) to create comparable data around the globe for assess- ing countries’ construction permit systems. Limited evidence exists directly verifying the effects of reforms changing construction permit regulations for warehouse construction. It is unclear how well the construction of a ware- house represents other aspects of planning regulation (for example, con- cerning retail and office buildings). The literature review on DWCP highlights the issue of the methodology de- pending on a highly stylized case study, as it has become unclear how rep- resentative it is of the requirements of most modern businesses across the globe. Therefore, it is unclear how useful the case study might be to capture broader aspects of planning regulation. The review suggests that indicator extensions could consider drawing comparisons with other common building types “such as retail and office buildings, often required by modern business- es” (CITATION, PG#). Likewise, the Independent Evaluation Group’s (IEG’s) desk review of literature identifies studies “indicating that the time and cost of getting construction permits are not correlated with economic growth and that the indicator itself does not align with the experience of surveyed firms” (CITATION, PG#). The DWCP portfolio included 121 projects and 167 interventions, adding up to US$304 million. For this indicator, the distribution of projects is balanced between the three types of advisory services: World Bank advisory services and analytics (ASA), World Bank reimbursable advisory services, and In- ternational Finance Corporation advisory services (IFC AS). For the period 158 in consideration, advisory services represent most Bank Group actions (77 percent) related to DWCP. The advisory services combined portfolio amounts to US$24 million. Advisory projects represent 8 percent of the total amount of resources dedicated to the area considered in the analysis, but they represent nearly 80 percent of total projects and interventions related to the indicator. On average, financing projects associated with DWCP have remained the same over the last decade (three lending projects per year). On the other hand, nonlending services show a constant growth in the previous decade. Regarding country reforms related to this indicator, most of them can be considered as positive regulatory changes (80 per- cent). Those positive reforms tend to concentrate in low-middle- and upper-middle-income countries. It has been a relevant tool for some country actors as an entry point for pro- moting sectoral reforms enhancing the efficiency of the construction sector. Some stakeholders suggested it would be better for DWCP if DB would only ascribe scores without explicit rankings or cluster countries by tier groups contingent on their score. Also, in terms of public policy decisions, DWCP scores between cities (where available) seem to be a positive factor enhanc- ing local reforms and promoting competition within countries. In general, DWCP interventions are effective. Areas with higher rates of success are “raising support for and awareness of reform” (100 percent, n = 3), “streamlined procedures” (86 percent, n = 74), and “improve business laws Independent Evaluation Group World Bank Group    159 or regulations” (83 percent, n = 115). Among the key factors that influenced project outcomes were analytical work, design complexity, proactive client engagement, effective coordination with partners and donors, overall M&E considerations, client commitment, and collaboration with external donors. External and internal outcomes that facilitated implementation and led to positive results (rated as adequate) include “effective coordination with part- ners or donors,” “analytical work,” and “collaboration with external actors or donors.” Overall, social and environmental incidence is weakly associated with DWCP projects; only 30 percent of the evaluated projects considered social and environmental goals as part of their objectives. The analysis of the distribution of objectives for DWCP-related projects reveals a missed opportunity for addressing associated process risks, corrup- tion, and informality/evasion of rules. As noted in the literature review and further in the stakeholder analysis, the construction sector is mentioned as one of the most corrupt business areas, with important challenges regarding informality and rule compliance. An examination of the objectives for DWCP projects shows that countries might be focusing on projects that improve a score in the DWCP ranking, rather than those needed to overcome persistent industry obstacles. In this regard, the World Bank Group could play an active role by adapting its measurements to create new incentives for projects as- sociated with those challenging areas in the construction sector. Regarding quality control, World Bank analysis shows that economies with a more effi- cient construction permitting system also tend to have better quality control The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix D and safety mechanisms. An example of how key regulatory implementations can help the DWCP area is Macedonia, where the number of procedures and time to build a warehouse were considerably reduced following the introduc- tion of a private third-party building plan review. Finally, from the stakeholder analysis, some general conclusions about DWCP can be drawn. Consistent with the literature review, the warehouse model fails to capture broader aspects of construction permits for many countries. This criticism suggests that the indicator might be inappropriate in some contexts and major cities with established industrial areas, such as Beijing and Moscow. For example, in Russia, because enterprises seek rele- vant information to guide their business decisions, the indicator limitation has led them to develop a separate roadmap only for industrial facilities. Through this parallel roadmap, dialogue between the Ministry of Construc- tion and the Russian Union of Entrepreneurs has taken place. The indicator presented some measurement concerns in Russia, as finding a typical ware- house that corresponded to the case considered in its methodology was chal- lenging. Consequently, the warehouse model was abandoned and substituted by a housing model. An additional methodological limitation of the indicator is the high level of knowledge required to answer the questionnaire, which is hard even for experts in the construction sector. Also, the DWCP indicator is seen as one of the most complex DB measures since many agencies from dif- ferent government levels get involved. Deeper scrutiny is needed, but a pre- liminary overview suggests that DWCP could be one of the less cost-efficient indicators in the DB report. Additional shortcomings to consider about the 160 DWCP indicator include the absence of consideration of additional measures for fire safety, the use of nonconvenient (potentially hazardous) materials for construction, country geological conditions, and informality. Getting Credit The DB getting credit (GC) indicator measures two things considered to be crucial to a country’s financial infrastructure: the legal rights of borrowers and lenders (strength of legal rights index) and the availability of credit information about firms and individuals (depth of credit information index). The strength of legal rights index measures how well collateral and bank- ruptcy laws facilitate lending. It assigns a country one point for each of ten attributes of collateral law and two for attributes of bankruptcy law, based on information provided by financial lawyers. The depth of credit information index measures the quality, scope, and accessibility of credit information through public and private credit registries. The data are derived from bank- ing supervision authorities and credit registries. Additionally, DB gathers data on the share of the population covered by public credit registries and private bureaus, but these data are not included in the calculation of the overall ease of doing business index. The GC indicator does not weight each of the 20 subindicators equally, which, for example, means that the collateral law is five times as important Independent Evaluation Group World Bank Group    161 as the bankruptcy law in terms of legal rights. Furthermore, the indicator appears to be geared toward consumer credit and not small and medium enterprises or corporate credit, thereby making it somewhat unreflective of access to finance for firms. On the legal rights aspect of the indicator, ex- perts revealed that the first point related to the existence of an integrated or unified legal framework which, despite shortcomings in the wording and interpretation, is perhaps the most important element and follows interna- tional best practices. Empirical evidence identified in the structured literature review on the effects of reforms strengthening credit reporting systems and collateral laws is mixed, but it does support the idea of increasing lending and improving borrowers’ performance. There was notable reform activity with 371 reforms recorded in 154 countries in DB reports between 2010 and 2020. By country, Rwanda was the top reformer with seven positive reforms recorded during this period. In 2010, Rwanda strengthened its secured transactions system by allowing a wider range of assets to be used as collateral, permitting a gener- al description of debts and obligations in the security agreement, allowing out-of-court enforcement of collateral, granting secured creditors absolute priority within bankruptcy, and creating a new collateral registry. Other top reformers included Kenya, Vietnam, Indonesia, Azerbaijan, Mauritania, and the United Arab Emirates. Over the evaluation period, the World Bank Group GC-related portfolio comprised 126 operations with a total of 195 interventions and US$1.348 The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix D billion committed. Most of these (71 percent) were advisory projects togeth- er valued at US$45 million. IFC AS projects represented just under half of the portfolio (42 percent), while World Bank nonreimbursable ASAs and reim- bursable advisory services accounted for about a third (30 percent) of oper- ations. Lending operations came only from the World Bank and represented the lowest share of projects (29 percent). World Bank GC-related projects were most prevalent in lower-middle-in- come countries, with 46 projects approved over the evaluation period, representing 39 percent of the portfolio. This was followed by upper-mid- dle-income and then low-income countries with 31 percent and 21 percent respectively. High-income countries accounted for 9 percent. However, while World Bank lending and IFC AS projects were mostly in lower-middle-in- come countries, ASAs were mostly used in upper-middle-income countries (45 percent). In terms of relevance, this indicator remains very influential to motivate government interest in carrying out legal and regulatory reforms related to secured transactions and credit information systems. However, some shortcomings in the indicator constrain relevance. For example, the depth of credit information index does not distinguish countries with both credit bureaus and credit registries from those with only a credit registry, despite their increasingly different functions. Other exogenous factors can affect ac- cess to finance beyond what is captured by the GC indicator. For example, an Ernst and Young analysis regarding the effect of sanctions, based on a survey 162 of 95 firms on the ease of doing business in Russia, showed that 27 percent have found it more difficult to finance Russian projects. The usefulness of the GC indicator in motivating government interest in embarking on reforms to their credit infrastructure was sometimes over- shadowed by the interest in improving rankings. Project design has been sometimes driven by client interest in simply improving the rankings, which limited the depth of reforms. In some instances, the minimum possible re- forms were carried out to improve rankings. The GC-related DB operations supported by the World Bank Group and evaluated by IEG were mostly rated as successful. World Bank lending and development policy operation projects had the larger share of successful interventions at 70 percent, while investment project financing projects had the lowest share of successful interventions at 50 percent. However, IFC in- terventions had a lower success rate, as about half of advisory interventions (54 percent) were rated as successful. The most successful interventions were targeted at streamlining procedures and raising support for and awareness of reforms. The highest number of in- terventions (19) was aimed at improving business laws or regulations, espe- cially with regard to secured transactions and credit information systems. An example of such was in Pakistan, where a 2016 approved World Bank devel- opment policy operation (P157207) supported an amendment to the Finan- Independent Evaluation Group World Bank Group    163 cial Institutions Bill that facilitated the use of inventories and receivables as collateral and strengthened lenders’ claims on collateral. Regarding immediate outcomes, successful interventions aimed to stream- line procedures; however, there was limited success in realizing intermediate outcomes in GC. Other successful interventions also included those that aimed to improve administrative systems (75 percent), while interventions to improve collateral laws and credit information systems had moderate success (67 percent). Limited success in GC was realized in interventions that aimed to reduce the cost of doing business, for which just over half of interventions were successful (56 percent). This result raises concerns about the overall effectiveness of interventions in this area, in realizing important outcomes for stakeholders such as increased lending. Factors that facilitated effectiveness and outcomes of World Bank Group GC interventions included adequate use of analytical work in project design and identification of risks at appraisal. However, at the design stage, effective- ness was constrained when projects were not designed to suit the client’s capacity and consider political and institutional capacity for reforms. At the project supervision stage, proactive client engagement and follow-up are examples of internal factors that facilitated implementation, effectiveness, and outcomes in GC. Expert interviews also revealed the presence of parallel systems in practice as a factor that explains the limited benefits of reforms. They pointed to the fact that in certain cases, countries pass legislation and develop collateral registry systems in line with DB to get score points on the The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix D indicator but have alternative systems in practice. Suggestions to improve the indicator’s relevance include recognizing the differences between countries with credit bureaus and credit registries from those with just public credit registries, recognizing the use of credit scores, adapting to capture rapid technological changes, and considering queries received. Experts raised several suggestions to improve the relevance of the indicator. For example, in recognizing the use of credit scores, experts have suggested incorporating a question that checks whether the bureau/registry produces scores. The need for the indicator to be adapted to capture rapid technological changes was also highlighted in expert discussions, given the data-intensive nature of credit infrastructure systems and the advent of al- ternative forms of data. For example, in advanced economies, credit bureaus should also consider financial transactions via mobile payments and media subscriptions (for example, Netflix). Key lessons from the evaluation period include identifying a champion to lead the project, with regulators (especially central banks) being one of the most interested stakeholders in embarking on reforms in this area. Other lessons include agreeing on a clear medium-term strategy, pursuing clear and consistent communication between the project team and the regulator, and relying on best practices. Finally, the deep dive highlighted the need to leverage technology in the adoption of new products, given rapid technolog- ical changes in credit infrastructure systems; create incentives to use newly established credit registries; and link the registry with other systems already 164 in place. Paying Taxes Introduced in 2006, the DB paying taxes (PT) indicator aims to record the cost of taxes and mandatory contributions a medium-size company must pay each year, as well as the firm’s administrative burden of paying them. It com- prises four subindicators that can be classified under two main components, tax policy and tax administration. Under tax policy is the subindicator (i) total tax and contribution rate, and under tax administration are (ii) tax pay- ments, (iii) time, and (iv) postfiling index (added in DB2017). The subindi- cators are aggregated in an overall score, computed as the simple average of the scores for each of the two component indicators, with a threshold and a nonlinear transformation applied to the total tax and contribution rate. The tax policy component has raised concerns regarding its measurement of the businesses’ effective rate. The indicator leads to a focus on the legal incidence of taxes (who writes the check to the government), instead of the more relevant economic incidence (who bears the financial burden), which depends on the relative elasticities of supply and demand. The model case also does not consider tax policy instruments that influence firms’ decisions and impact their final tax rate (for example, tax rates segmented by indus- try or size, tax subsidies, special economic zones, and other tax incentives and exceptions). In Jordan, for instance, the PT indicator did not capture the expansion in tax exemptions and zero-rated commodities introduced in Independent Evaluation Group World Bank Group    165 2008 and its consequences for tax integrity, revenue base, and taxpayer and administrative compliance. Additionally, it imposes a judgment about the social preferences embodied in what it measures. Tax rates reflect social preferences and other factors embodied in fiscal policy. The rating essentially penalizes a country for making potentially smart choices reflecting social preferences about fiscal policy (irrespective of how well fiscal resources are used). This was observed by previous assessments, as the component goes beyond the measurement of compliance costs, reflecting countries’ overall fiscal policy decisions derived from social preferences, enforcement and collection capacity constraints, and political considerations. Hence, the component does not allow for mean- ingful benchmarking across economies. While experts have observed the general adequacy of the indicator regard- ing tax administration, there are unaddressed challenges, especially when it comes to digitization. Experts and users of the PT indicator have noted that the indicators are a simple and fairly complete snapshot measure of firms’ administrative burden of complying with tax obligations. The inclusion of the postfiling subindicator in DB2017 was welcomed; nonetheless, tax administration trends are changing rapidly with the surge of new technolo- gies and availability of information. The PT indicator does not fully take into account digitization in shortening time to pay taxes. More importantly, it does not incorporate emerging digital solutions that are disrupting the way tax administration is done globally (for example, compliance-by-design and The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix D third-party-collection approaches made available by technology). With a total of 531 reforms related to PT tracked in the reports published from 2010 to 2020, it was one of the most reformed DB areas. While almost all countries had at least one PT reform (170), the low-income group was the least frequently reformed (14 percent). The reforms were related to tax pol- icy (60 percent) or tax administration (40 percent). The Europe and Central Asia region was the one with most reforms (36 percent), followed by Africa (21 percent). PT is also the business area with most negative reforms tracked (after labor regulations/employing workers). Tracked PT reforms were mainly policy focused and less frequently took place in low-income countries. The DB team tracked 531 national-level PT reforms in 170 countries, making it the second-most-reformed business area after starting a business. The reforms were related to tax policy (60 percent) or tax administration (40 percent). The Europe and Central Asia region was the one with most reforms (36 percent), followed by Africa (21 percent). The country with the most tracked reforms was Vietnam, with eight, followed by Indonesia, Hungary, and Brunei Darussalam. High-income countries were the most frequently reformed. PT is also the business area with most neg- ative reforms tracked (after labor regulations). While PT reforms related to the administrative component were generally positive (95 percent), the ones related to the policy component were 46 percent negative. In PT-related reforms supported by the World Bank, the DB indicators were mainly used as subsidiary elements in the justification for the projects. Al- 166 most half of the approved operations and slightly over half of the approved interventions used DB to justify the operation. At the same time, over a quarter of both projects and interventions used them as a results indicator. The DB PT indicator was mainly relevant for the World Bank to open a channel of dialogue with governments. IEG conducted interviews with tax specialists and government officials that yielded examples of the role that the PT indicator has played in relevant reform processes: warning leaders of the executive branch of their national tax policy and administration lim- itations. In some cases, the PT ranking was the country’s key driver of the dialogue and reform (for example, in Kazakhstan). In others, the ranking improvement was expected to be a side effect of it (for example, Armenia or Vietnam). It is worth mentioning that tax experts from other multilateral organizations (namely the International Monetary Fund and the Inter-Amer- ican Development Bank) stated that this beneficial effect (improvement in rank) was not perceptible in their projects. Another use of the DB indicators has been to communicate policy improvements to constituents. The histo- ry of the DB report helps explain why its indicators became a main tool for official communication in some countries. The simplicity of its ranking logic was instrumental when communicating targets more broadly. An example of it is Russia, which by 2012, when occupying the 120th rank of 189 overall in ease of doing business, approved a presidential decree with ranking targets of attaining 50th place by 2015 and 20th by 2020. Independent Evaluation Group World Bank Group    167 Despite the strong criticisms of the policy component of the PT indicator po- tentially incentivizing countries to seek improvements of DB performance by decreasing tax rates (a race to the bottom), government officials interviewed did not report to have been affected by that pressure. Yet the incentives in place allow for countries, in trying to position themselves internationally in the DB rankings, to cut tax rates, potentially making suboptimal policy deci- sions and even undermining their capacity to finance basic public services. Having said that, government officials interviewed did not report feeling pressured to make policy-related reforms to improve the county’s score. When there was enough state capacity to follow smart policy decisions, the policy aspect of the PT indicator’s composition did not lead to pressure for different tax-level decisions. There is some disconnect between the PT indicator and what businesses report. Notwithstanding Russia’s successful experience in improving the indicator’s ranking, moving from 103rd in 2010 to 58th in 2020, the Business Environment and Enterprise Performance Survey shows that enterprises considered corporate tax rates their biggest constraint in both years. In Co- lombia, the indicator was in fact used as evidence that corporate tax rates are high (the country ranked 148th in DB2020 with a total tax contribution rate of 71.2 percent), and there were reforms including tax cuts during the period. However, government officials reported that while DB was a tool to support such a shift, tax rate decisions and reforms were not based on DB alone. Best practices found evidence of good use of the indicator when the coun- The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix D try understands the limitations of the indicator, both in terms of policy and of its potential for design, and has public and private digital technology capacity developed to adopt new features. The indicator did undergo some changes feeding back from experts’ critics. A first lesson was that stakehold- ers of the DB PT indicator should understand its limitations when using it. In the World Bank Group, operations often use the PT indicator as a M&E tool, and that can become an issue when improving the indicator becomes a priority over improving real outcomes of the system. In the countries, it is important for authorities to understand that decreasing corporate taxes should not be used as a fast-track, easy-to-implement way to earn score improvements in the PT ranking. In Colombia, authorities were aware of the changes that would influence rankings, but this was not the only mo- tivating factor. In Georgia, where the ranking was the main goal, the social and economic consequences in terms of the government’s ability to finance public services were reportedly negative. When designing reforms, countries should and often do combine different sources of information along with the PT indicator to make policy and administrative decisions. Using other indi- cators (for example, the Tax Administration Diagnostic Assessment Tool) may improve the intervention’s design, as that is the PT’s main deficiency. A second lesson is that digital technologies, automation of data collection, and improved taxation by design are growing trends, and countries that count on widespread internet connection may be ahead in their capacity to improve tax administration and avoid tax evasion. The PT indicator’s current design does not account for those trends. 168 Regarding indicator improvements, the PT indicator did undergo design changes to its policy component to address expert feedback received, includ- ing the addition of postfiling performance. DB also established a minimum rate threshold, to respond to the criticism regarding incentives to decrease tax rates leading to zero, making small tax-free islands top performers. The changes were not meaningful for most countries, however, since the thresh- old rate below which countries were not penalized by the indicator was set very low. Starting a Business The starting a business (SB) indicator measures the burden of opening a lim- ited liability company, including for women-led businesses. The SB indicator focuses on the number of procedures, days to register, and the costs of regis- tration required to formalize a business, with a specific measurement of that from women-led businesses. A number of assumptions are made for the base case, such as engaging in domestic business activities without importing or exporting, and having five national “sane” shareholders with no criminal re- cords that are monogamously married. This indicator has been shown to be a significant motivating factor for reform, and it has featured prominently in World Bank Group assistance, both financial and advisory, for its developing member countries. Some 17 percent of all Bank Group interventions involv- ing the DB indicators have featured reforms related to SB, one of the highest Independent Evaluation Group World Bank Group    169 percentages among the DB business areas. The DB report claims that the SB indicator captures public policy objec- tives related to promoting private sector development. The indicator has influenced policy makers’ perceptions regarding the need for reforming the framework for starting a business. In this sense SB has played an important role among the whole set of indicators. The framework for business creation and operation is of central importance for the business environment that supports private sector development, but there are important limitations to it. The narrow focus of the SB indicator does not capture essential elements of the legal and regulatory processes that are part of the institutional framework that relates to how businesses are started, how they operate, and how they are closed. Furthermore, SB does not provide sufficient information to identify reform priorities beyond giving a very general indication based on the number of days and costs to register a business and how this relates to other countries. In many cases SB issues are not significant constraints and do not reflect far deeper problems with the legal framework for business operation. Despite some of them not being considered in the SB indicator, laws sur- rounding business creation and operation are central to the institutional framework for private sector development. The simple measures that com- pose the SB indicator are far too general to provide guidance on how to reform, although they could indicate that reform might be necessary. Sim- The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix D ply reducing costs and steps in business formation in order to improve the ranking of the SB may make the reforms ineffective. The components of the SB indicator that are used to calculate the rankings of the ease of starting a business are also widely disparate in their importance. For example, elim- inating the need for a company seal is equally rewarded in terms of reform effectiveness as reducing the number of agencies with which an entrepre- neur must register. This focus has led to reforms inspired by SB frequently emphasizing the establishment of one-stop shops to reduce the time, cost, and number of procedures involved in starting a business despite evidence of their effec- tiveness being weak. A further methodological issue is whether it is a real- istic assumption that entrepreneurs undertake all the steps themselves and whether it reflects the actual situation facing an entrepreneur. In most coun- tries where registration processes are complicated, lawyers, accountants, and professional “fixers” provide this service for a fee. While the SB indicator has been shown to motivate reform, there is little evidence that a higher SB ranking necessarily means that the framework for starting a business has improved, or that there have been more business registrations. In addition, the “typical” company on which the analysis of SB is based seems to be unrealistic. The SB criteria require that this typical company have five shareholders, either all men or all women, each with an equal shareholding. A further requirement is that the typical company neither exports nor imports, which is at odds with the trading across borders compo- 170 nent of the indicators. At the same time, much of the focus of the SB indi- cator appears to be on formalizing low-productivity, informal businesses. Research has shown that these businesses fail at a disproportionately high rate, in contrast to businesses that formalize without a period in the infor- mal sector. Another important issue, which the SB indicator does not address, is how the costs of formalization relate to the benefits of being formal. Formalization subjects businesses to taxation and regulations that raise the cost of doing business. If these costs are not offset by benefits, including having access to finance, being able to use the legal system, and being able to engage in cross-border trade, incentives to formalize are weak. The very general nature of the SB indicator provides little indication of exactly how to reform the SB framework. To achieve far-reaching reforms, in-depth analysis of the laws and regulations governing the establishment of different types of businesses is an essential step in developing a reform strategy. This involves drafting, passing, and implementing meaningful legislation and associated regulations that relate to the various types of business organization, such as sole proprietorships, limited liability cor- porations, and public companies. The process is complex and can take an extended period to come to fruition. Furthermore, there is no “one size fits all” type of legislation. Particularly in developing countries, such legislation must be adapted to specific country circumstances. None of this essential Independent Evaluation Group World Bank Group    171 information is reflected in the SB indicator. There is limited evidence that the indicator has promoted necessary changes in the legal and institutional framework governing businesses, beyond perhaps identifying that reform is necessary. An open question is whether SB is focusing on the wrong priori- ties—eliminating procedures and costs may well have little impact on pover- ty alleviation and growth. While there is a paucity of evidence that shows the beneficial impact of im- proving the SB ranking, there is evidence that SB fails to reward fundamental reform. Case studies on the framework for SB in Colombia and Pacific Island countries show that deep-seated reforms of the institutional framework for business formation and operation lead to longer-term sustainable busi- ness creation but that these have not been reflected in the rankings, which disproportionally reward the establishment of one-stop shops. Reforms in Rwanda and Timor-Leste involving one-stop shops have led them to being ranked among the leading reformers in the world. However, there is little evidence that these reforms led to a higher rate of business creation. It is therefore highly uncertain that an improvement in the SB indicator leads to an improvement in the business environment. Trading across Borders The DB indicator for trading across borders (TAB) measures the time and cost associated with the logistical process of exporting and importing a stan- The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix D dardized shipment. It is assumed that each economy imports a standardized shipment of 15 metric tons of containerized auto parts (worth approximately $50,000) from its natural import partner—the economy from which it im- ports the largest value (price times quantity) of auto parts. It is also assumed that each economy exports the product of its comparative advantage (de- fined by the largest export value) to its natural export partner—the economy that is the largest purchaser of this product. For example, for Armenia, their hypothetical export case is based on the export of “beverages, spirits and vinegar” to the Russian Federation, and the hypothetical import is auto parts from Russia, its leading trade partner. IEG’s prior structured literature review finds that the TAB indicator captures some of the less directly observed trade cost elements. It finds that TAB and the indicators of the separate Logistics Performance Index have the benefit of broad coverage and comparability across countries and are thus used in a large number of research studies on trade facilitation. Nonetheless, these metrics have inherent limitations. For instance, the TAB indicator relies on the responses of freight forwarders and, in a smaller proportion, customs and ports authorities. In the same manner, the Logistics Performance Index is based on perception scores given by freight forwarders and express carriers (World Bank 2016). As such, these metrics are not as free from cultural biases of the respondents compared with sources like information from adminis- trative records. Finally, the two sets of indicators likewise go through regular refinements that attempt to improve sample representativeness and the 172 coverage of the indicators (World Bank 2016, 2017). Hence, users ought to exercise care in ensuring comparability across time. Studies also show that the DB TAB indicator is positively correlated with countries’ export performance (for example, see Iwanow and Kirkpatrick 2009). In addition, several studies indicate that faster transit times are cor- related with increased trade (Djankov et al. 2010; Hoekman and Nicita 2011; Rocha 2011) and lower levels of trade-related corruption (Shepherd 2009), and that countries with quicker export times are more likely to export more time-sensitive goods (Li 2019). TAB cannot always capture the degree of implementation. Frequently, re- forms implemented in the main port of the main city of a country take far longer to be implemented elsewhere. Sometimes, DB picks up reforms when enacted as regulations rather than when implemented. Further, the hypo- thetical case structure can miss a lot and cover only a small percentage of total trade. It is hugely sensitive to changes affecting the hypothetical case. For example, after Armenia joined a customs union with Russia, its DB TAB ranking jumped from 110th to 29th in a single year based on the focus of the hypothetical case. Finally, TAB measures only compliance costs and the speed of procedures, ignoring whether trade regulations are achieving their objectives (that is, detection rates, revenue collection, public health and safety, and so on). Independent Evaluation Group World Bank Group    173 Support for TAB reforms has been quite common in the Bank Group portfo- lio, even though World Bank lending tailed off in the latter part of the period. While encompassing elements of 195 projects, lending slowly dropped and advisory support remained steady. TAB is used for multiple purposes, but most commonly for process re- engineering (simplification, harmonization, and streamlining), improving business laws and regulations, conducting diagnostics, formulating new regulatory strategies and policies, or capacity building. While World Bank lending and IFC AS most often focused on reengineering processes and improving business laws and regulations, World Bank ASA and reimbursable advisory services often supported diagnostics and new business regulation strategies or policies. This suggests a complementarity of instruments. IEG’s trade facilitation evaluation (2019) found that the DB indicators were generally very motivating for reforms and generally understood by clients but did not provide granular enough information to guide the design of reforms or monitor their progress and outcomes. Common interventions require a detailed mapping of procedures and identification of sources of de- lay or excess cost. They should involve attention to managing risks of non- compliance with border regulations, such as adverse environmental, health, social, and security outcomes. The Bank Group and clients need to generate and monitor accurate and timely data to identify and address performance challenges and to measure outcomes. TAB-informed projects in the Bank Group portfolio were generally suc- The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix D cessful. Institutionally, World Bank lending had the highest success rate at 83 percent, with development policy lending (86 percent) considerably outperforming investment lending (67 percent). IFC’s advisory services succeeded 71 percent of the time. Project success paralleled country income levels, with a 63 percent success rate in low-income countries compared with 79 percent in lower-middle-income and 82 percent in upper-middle-income countries. Regionally, the lowest success rate was in Sub-Saharan Africa. By objectives, efforts to improve laws, procedures, policies, and strategies were the most successful, while efforts to streamline procedures were distinctly less successful. Data were not collected consistently across projects that would allow comparison of outcomes. Among 20 project evaluations that examined data on trade flows, 15 percent (3) showed a positive outcome. One of only three projects examining costs to businesses found that inter- ventions lowered them. Nevertheless, much like other DB indicators, TAB cannot reflect the impor- tance of trade in an economy nor the priority of trade facilitation reforms relative to other constraints. Further, it is focused primarily on simplification of rules and procedures, leaving out many other aspects of trade facilitation reforms and logistics. It only sometimes catches reforms and thus is a weak metric for reform activity. For example, it reflects only two of Laos’s multiple trade facilitation reforms (DB2008 and DB2013). IEG’s econometric analysis shows a poor fit between Bank Group–supported reform activity and most DB TAB indicators. On top of this, its changes in methodology have created 174 substantial discontinuities in apparent country performance in some cases. IEG’s trade facilitation evaluation (2019) found significant and beneficial relationships between Bank Group trade facilitation interventions and outcomes as measured by the time to export and time to import TAB sub- indicators. It did not find a significant relationship between Bank Group trade facilitation interventions and TAB subindicators measuring the cost to import and export. That analysis suggested different comparative advantag- es of IFC and the World Bank—with IFC especially effective at supporting im- provements in border infrastructure, and the World Bank especially effective at supporting improved border function and technology, building capacity of border agencies, and supporting improvements in rules. The same evaluation found clear synergies between the different areas of trade facilitation reform, except border infrastructure. It advocated a more systematic approach combining simultaneous or sequential interventions in several areas of trade facilitation reforms and combinations of various Bank Group instruments. Yet it found that “few Bank Group projects reference the adoption of a programmatic approach to trade facilitation reforms.” It stated that the Bank Group, “recognizing its resource constraints, can optimize by coordinating with other donors and giving priority to clients where trade facilitation is a priority and where [the Bank Group] has comparative advan- tages” (CITATION, PG#). External factors did not figure as prominently in explaining individual proj- ect outcomes as did factors under the Bank Group’s direct control. Quality at Independent Evaluation Group World Bank Group    175 entry was the leading factor for development policy loans; M&E and results frameworks were the most common success factors for investment projects; and project supervision figured most prominently in explaining outcomes of IFC AS. At project entry, prior analytic work was a key quality factor for success, as was the suitability of project design to client capacity. For project supervision, two leading factors were proactive engagement and follow-up with the client and effective coordination with other development partners and donors. M&E factors included how (and whether) the monitoring system was implemented, as well as the design and utilization of the M&E system. Not surprisingly, the two external factors that mattered the most common- ly for success were client commitment and agency coordination, including political economy aspects. 176 The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix D Table D.1. Summary of Case Studies and Deep Dives Evaluation Findings from Case Studies and Question Line of Inquiry Explicit Hypotheses Deep Dives Recommendations 1. What is the Whether all DB indi- 1. DB indicators vary in their » Many stakeholders reported that Linked to the following findings: relevance of DB cators are relevant to relevance to the key con- while they found DB useful in » Experience shows many in- indicators to constraints influencing straints influencing busi- informing, supporting, and moti- stances where indicators are a. The business business dynamics and ness dynamics, hence the vating aspects of country reform not granular enough to reflect environment economic contribution economic contribution of priorities, DB does not capture key reform progress that is devel- priorities of client of policy reforms. reforms, based on the area components needed in structural opmentally effective. countries? and the country context. reforms, focusing on a fixed agen- • It is desirable for the DB da of legal and regulatory issues approach to capture a fuller b. The substan- often complementary to critical range of regulatory, legal, tive dimensions priorities. and institutional conditions within each » In some countries, such as China that influence the life cycle of indicator’s focus after 2017, DB had an enormous enterprises. area? influence on reform plans and » Currently, indicators suffer from c. World Bank activities of some central and inadequate feedback loops Group strategic local authorities. In other cases, between their design and ap- and operational such as Russia, some of the main plication on the one hand, and priorities? country constraints (corruption, research and field experience governance, rule of law, better on the other. labor market allocation) were not addressed by DB, and hence it had limited effect on business dynamics. » The trading across borders indica- tor was generally very motivating and understood by clients but was not granular enough to guide the design of reforms or monitor their progress and outcomes. (continued) Evaluation Findings from Case Studies and Question Line of Inquiry Explicit Hypotheses Deep Dives Recommendations . and Whether (i) client 2. Client institutional capac- » Even simple reforms may require » DB methodology and base institutional capacity ity and capability is a key interministerial coordination and case scenario may not neces- and ability (high vs. low) factor influencing how DB high-level oversight, and coordi- sarily reflect local conditions. to tailor the use of DB indicators should be used nation between central and local The Bank Group needs to indicators to its devel- and whether their use is governments. provide such context in its opment priorities and effective. » In countries such as Rwanda, guidance, not only in textual framework and (ii) client DB-informed projects were delayed caveats in DB reports, but at motivation (for example, due to lack of sufficient capability all stages of engagement, in to achieve policy reform of project implementation units. In part by tailoring guidance to vs. improve DB ranking) Indonesia, institutional capacity was local context and drawing from are key influences on also mentioned as one of the factors multiple sources of evidence the use and appropri- affecting project implementation. and analytic frameworks. ateness of DB indicators. » In some countries, leaders were mo- tivated more by appealing to donors and foreign investors than by placing a priority on SME dynamics. » Dealing with construction permits is seen as one of the most complex DB measures since many agencies from different government levels get involved, and its accurate imple- mentation involves capacities not easy to find even in experts in the construction sector. » Paying taxes requires progressively more sophisticated steps, requiring deeper technical capacity to design and implement. (continued) Independent Evaluation Group World Bank Group    177 178 The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix D Evaluation Findings from Case Studies and Question Line of Inquiry Explicit Hypotheses Deep Dives Recommendations 2b. Are clients us- 3. Client motivation is a key » The importance of client motivation » In line with existing good prac- ing DB to achieve factor influencing how DB for reform success was evident tice, continue using DB to mo- improved out- indicators should be used in China, Russia, India, Indonesia, tivate client engagement and comes for busi- and whether their use is Rwanda, and Morocco. to assist in focusing reforms ness environment effective. » DB has been very successful as an within DB’s menu of regula- reform in line with entry point in the policy dialogue tory areas, where the priority the country’s de- and a very effective tool to launch is confirmed by additional velopment policy reforms. analytics. priorities? 1a, 1b, 2b. Whether DB indicators 4. DB indicators are useful » DB indicators draw attention and » Continue using DB to motivate are useful to (i) draw to draw attention and resources to legal and regulatory client engagement on the attention and resources resources to legal and reg- reforms in the areas they measure, business environment. to business environ- ulatory reforms in the areas because rankings matter to coun- ment reform in areas they directly measure. try leaders. they measure directly, » Having high-level engagement (ii) draw attention and makes a difference, opening doors resources to business to reforms beyond the DB indica- environment reform tors. beyond the areas they measure directly, (iii) correctly identify and order priority reform areas, and (iv) monitor and evaluate reform implementation. (continued) Evaluation Findings from Case Studies and Question Line of Inquiry Explicit Hypotheses Deep Dives Recommendations 5. DB indicators are useful » Multiple country experiences con- » Indicates utility of building to motivate broader firm this, with a couple of excep- capacity and of using addi- processes of business tions. tional sources of evidence environment reform. » Often broader processes involve and frameworks to engage adaptation to country conditions in broader reform programs or replication of initial reforms addressing binding country in additional cities. Subnational constraints. DB reforms are an avenue often pursued. 6. DB indicators are useful » Overall, most countries have » The evaluation finds verification for identifying binding binding constraints that are not is needed before framing rec- constraints and priority captured by the DB indicators. ommendations solely based reforms. These include corruption, access on DB. Such reforms are lim- to other services (for example, ited in scope and depth, and water), competition challenges, may miss binding factors. and more. » The Independent Evaluation » In some cases, DB indicator cov- Group also encourages the erage (trading across borders and Development Economics paying taxes) does not capture the Vice Presidency to conduct real challenges experienced by research to fill gaps in the ev- SMEs. idence about the relationship of indicators to outcomes. (continued) Independent Evaluation Group World Bank Group    179 180 The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix D Evaluation Findings from Case Studies and Question Line of Inquiry Explicit Hypotheses Deep Dives Recommendations 7. DB indicators are useful » Due to their powerful motivat- » Discontinue using DB indica- to monitor and evaluate ing nature, many countries and tors as project indicators and reform progress. projects have included DB-rated objectives due to their lack of performance in national planning granularity and weak connec- and reform objectives. tion to project outcomes. » There are serious gaps between country movements in the indica- tors and project and development outcomes. » More sophisticated users, most International Finance Corpora- tion advisory services, and many World Bank projects develop more tailored indicators to better outcomes and to adapt to local context. 1a, 1b, 2a, 2b Whether complemen- 8. DB indicators are more » Good practice is to use other » Use additional evidence and tary data and guidance effectively used in con- indicators and analytic frameworks diagnostics to complement enhance the relevance junction with complemen- (for example, WEF, WGI, OECD, and contextualize DB. of DB indicators to tary data and guidance to UNCITRAL, TADAT), both to diag- business environment enhance the relevance, nose and to design solutions. reforms and the ability comprehensiveness, and » Additional sources of knowledge to follow with comple- depth of corresponding were mobilized in many DB-in- mentary or deepening business environment formed projects, including through reforms. reforms. peer-to-peer learning and global subject specialists. (continued) Evaluation Findings from Case Studies and Question Line of Inquiry Explicit Hypotheses Deep Dives Recommendations 2a. Is DB reform Whether the feedback 9. DB benefits from robust » Feedback loops are insufficient. » Strengthen learning and feed- achieving desired loops between available feedback loops between While indicator changes have back loops. outcomes? empirical evidence and empirical evidence (re- reflected some feedback, there is » Fill gaps in the evidence with a And derives from DB indicator design and search and evaluative data) a lot of evidence, lack of evidence, strategic research agenda. feedback loop in use are robust. and indicator design. and country experience that has » Update indicators in line with theory of change. not been addressed. a review of rigorous evidence, » Deep dives underscored the lim- feedback from M&E, and tech- ited relevance or validity of some nological changes in underly- indicators (for example, dealing ing processes. with construction permits and » The DB approach should cap- starting a business), as well as ture a fuller range of regulato- the adverse guidance they may ry, legal, and institutional con- provide (for example, paying taxes ditions that influence the life policy component). cycle of enterprises, to assure » Some country stakeholders crit- the framework includes and icized the inapplicability of base rewards the key attributes of a case scenarios to the real condi- legal and regulatory system for tions for firms in local context. SME development. » Deep dives and case studies pro- vide examples of where techno- logical change is not well reflected in DB indicators and where DB agenda loses relevance over time. (continued) Independent Evaluation Group World Bank Group    181 182 The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix D Evaluation Findings from Case Studies and Question Line of Inquiry Explicit Hypotheses Deep Dives Recommendations 1c. and derives Whether current insti- 10. Current institutional ar- » Stakeholders, unlike some Bank » This topic is addressed more from challenges tutional arrangements rangements separating DB Group staff, did not express a squarely by the process audit regarding data separating indicator indicator generation from perception of conflict of interest of the Group Internal Audit Vice integrity. generation from project DB reform project work within the Bank Group and were Presidency and by other inter- work (including reim- (including reimbursable often motivated to engage by nal Bank Group reviews. bursable assistance) are assistance) are adequate a perceived ability of the Bank adequate to prevent a to prevent a perception of Group to help move indicators. perception of conflict of conflict of interest within interest within the World the World Bank Group. Bank Group. Source: Independent Evaluation Group. Note: DB = Doing Business; M&E = monitoring and evaluation; OECD = Organisation for Economic Co-operation and Development; SME = small and medium enterprise; TADAT = Tax Administration Diagnostic Assessment Tool; UNCITRAL = United Nations Commission on International Trade Law; WEF = World Economic Forum; WGI = . References Djankov, et al. 2010. Hoekman, and Nicita, . 2011. Iwanow, and Kirkpatrick, . 2009. Li, . 2019. Rocha, . 2011. Shepherd, . 2009. World Bank. 2016. World Bank. 2017. Independent Evaluation Group World Bank Group    183 Appendix E. Desk Review of Literature As initially presented in this macro evaluation Issues Paper, the first liter- ature review conducted by IEG took stock of evidence on the DB effec- tiveness through a desk review of 426 articles shared with IEG by the DB team derived from 100 leading journals. The database was organized by indicator. Some areas had abundant coverage (starting a business, protecting minority investors, and trading across borders) but many others had much The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix E sparser coverage (e.g., dealing with construction permits and getting elec- tricity). The desk review did not assess the methodological rigor each arti- cle. Instead, IEG did that in a separate Structured Literature Review, which also cast a broad net to capture academic literature beyond the DB database shedding light on the relation of DB indicators and related reforms to out- comes (Annex F). IEG but did consider whether the article shed light on the relationship of what is measured by a DB indicator with an outcome, so its reporting is limited to 75 articles found to have strong relevance. The desk review identified relevant articles and, according to the fre- quency of the evidence found, classified the strength of the association found between DB indicators with specific outcomes as strong, modest, or mixed. The articles indeed related to the indicator and which specifically researched the Doing Business indicators were considered relevant. They were the sources of a listing of the evidence found on the association of all indicators with development and economic outcomes for businesses and economies. Findings confirmed by three or more articles was considered to have “strong evidence.” (table E.1) For example, there was strong evi- dence that improved trade facilitation is associated with increased trade flows. Most of the “modest evidence” findings were supported by a single article, such as the finding that Greater costs and procedures in enforcing contracts linked to lower economic growth. Where articles had contradict- ing findings, the evidence was classified as ‘mixed.” Much of this literature finds significant associations of DB indicator measures with outcomes, without necessarily establishing causation. 184 Table E.1. Summary of findings IEG Desk Review of DB Literature Database Indicator Strong Evidence Modest Evidence Mixed Evidence Starting a Business Higher entry costs Lowering entry cost Lower entry costs (79 papers) or more steps or associated with larger encourage formal- documents are SME sector, trade.2 ization. associated with Reduced com- less firm creation, plexity encourages growth or profit- entry. ability.1 High entry costs increase/ reduce productivity, em- ployment.3 Getting Credit Stronger legal More credit information Credit information (29 papers) rights associated associated with more sharing reduces with more lending, private credit, better the likelihood of more financial ac- lending terms, reduced banking crises.6 tivity, lower interest defaults/delinquen- rates.4 cies.5 Paying Taxes Higher tax rates Tax complexity, number Progressivity of tax (28 papers) constrain entrepre- of payments reduce rates encourages neurship, invest- formal entry, invest- entry.9 ment.7 ment, innovation.8 Enforcing Contracts Better contract enforce- (41 papers) ment encourages invest- ment in asset-specific (customized) inputs.10 Greater costs and pro- cedures in enforcing Independent Evaluation Group World Bank Group    185 contracts linked to lower economic growth.11 Lower contract enforce- ment costs reduce chance of early liquidation.12 Countries with more FDI and debt finance have lower contract enforce- ment costs.13 (continued) Indicator Strong Evidence Modest Evidence Mixed Evidence Trading Across Bor- Better trade facili- Countries with lengthy ders tation measured by export times are less likely (69 papers) DB (and LPI) shows to export more time-sensi- correlation with in- tive goods.15 creased trade flows.14 Faster customs clearance can offset the impact of being a landlocked or smaller country.16 Export gains from better trade facilitation outweigh gains of tariff cuts in im- porting African countries.17 Longer trade times are associated with higher The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix E levels of trade-related corruption.18 Construction Permits Time and cost of getting (17 papers) construction permits are not related to economic growth.19 No relationship between DB construction permit in- dicators and the presence of corruption/bribery.20 Construction permit indi- cators do not permit iden- tification of policies that matter systematically.21 Indicators do not align with the experience of surveyed firms, which re- port huge variation within countries.22 Registering Prop- Higher burden of regis- erty tering property associ- (22 papers) ated with reduced ben- efit of trade opening on national income.23 Registering property indicators have no rela- tion to FDI inflows into Sub-Saharan African countries.24 Registering property indicators are inconsis- tent with actual experi- ences of firms reported 186 in enterprise surveys.25 (continued) Indicator Strong Evidence Modest Evidence Mixed Evidence Protecting minority Country-level minority investors investors’ protection (60 papers) policies matter more than firm-level deci- sions for improving corporate governance.26 Lower levels of dis- closure discourage investment and affect corporate governance.27 Getting Electricity   Getting electricity   (16 papers) indicators have little predictive power for a country’s electrifi- cation rate and the level of technical and non-technical losses in the system.28 A better rank on ease of getting electricity is associated with greater perceived quality of electricity supply.29 Electricity connection processes tend to be lengthier and more cumbersome in coun- tries where other ad- ministrative processes are also burdensome.30 Independent Evaluation Group World Bank Group    187 Administrative reforms to reduce the number of days to get electricity can improve the rank- ings of the country.31 DB measures of elec- tricity constraints are negatively associated with measures of ener- gy consumption.32 Frequency of outages are correlated with difficulty of connection process, while elec- tricity tariffs are not associated with either measure.33 (continued) Indicator Strong Evidence Modest Evidence Mixed Evidence Resolving Insol- Longer time to resolve vency insolvency in some (34 papers) sectors is negatively associated to net job contribution by start- ups and, to a much lesser extent, by incum- bents.34 Inefficient bankruptcy is associated with less bond issuance by risky, but not by safe, borrow- ers.35 Legal infrastructure The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix E facilitating the reso- lution of insolvencies mitigates the negative effect that limited supply of long-term fi- nance has on economic volatility.36 DB’s recovery rate in- dicator correlated with availability of domestic credit from a country’s banking sector.37 Good insolvency regimes are negatively associated with stock of NPLs and their rate of increase.38 Good insolvency re- gimes associated with a lower frequency of insolvencies.39 (continued) 188 Indicator Strong Evidence Modest Evidence Mixed Evidence Employing Workers Developing Greater likelihood of (31 papers) countries with positive impact of Trade rigid employment liberalization on em- laws tend to have ployment and wages in larger informal countries with flexible sectors and higher labor markets and vice unemployment, versa.41 especially among More regulated labor young workers.40 markets tend to have higher wages at the expense of sector wide employment.42 There is an economically significant association between digital technol- ogy use by businesses and a country’s statutory minimum wage and its employment protection regulations.43 There is a negative correlation between the stringency of labor regu- lation and the intensity of its enforcement.44 Strong negative con- sequences of labor regu- lation on labor market outcomes, are based entirely on measures Independent Evaluation Group World Bank Group    189 of de jure stringency of regulations.45 Significant association between digital technol- ogy use by businesses and a country’s statutory minimum wage and employment protection regulations.46 Dismissal laws can prevent employers from arbitrarily discharging employees and thereby enhance employees’ innovative efforts and encourage firms to invest in technology and innovation projects.47 Source: Independent Evaluation Group. References Acar, Aysenur, Laurent Loic Yves Bossavie, and Mattia Makovec. (2019). Do Firms Exit the Formal Economy after a Minimum Wage Hike? Quasi-Experimental Evidence from Turkey. Policy Research Working Paper 8749, World Bank Acharya, Viral V., Ramin P. Baghai, and Krishnamurthy V. Subramanian. (2013). Labor Laws and Innovation. Journal of Law and Economics 56(4): 997–1037 Ahlquist, J. S. and Prakash, A. (2010). FDI and the costs of contract enforcement in developing countries. Policy Sciences, 43(2), 181-200 Andersson, M. and Henrekson, M. (2015). Local competitiveness fostered through The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix E local institutions for entrepreneurship. The Oxford handbook of local competi- tiveness, 145-190. Arlet, J. N. (2017). Electricity sector constraints for firms across economies: a com- parative analysis (No. 117320, pp. 1-4). The World Bank. Athari, S. A., Adaoglu, C., and Bektas, E. (2016). Investor protection and dividend policy: The case of Islamic and conventional banks. Emerging Markets Review, 27, 100-117. Ayyagari, M., Beck, T., and Demirguc-Kunt, A. (2007). Small and medium enterprises across the globe. Small business economics, 29(4), 415-434. Baliamoune-Lutz, M. and Garello, P. (2014). Tax structure and entrepreneurship. Small Business Economics, 42(1), 165-190. Becker, B. and Josephson, J. (2016). Insolvency resolution and the missing high-yield bond markets. The Review of Financial Studies, 29(10), 2814-2849. Belitski, M., Chowdhury, F., and Desai, S. (2016). Taxes, corruption, and entry. Small Business Economics, 47(1), 201-216 Block, J. (2016). Corporate income taxes and entrepreneurship. IZA World of Labor. Boedo, H. J. M. and Mukoyama, T. (2012). Evaluating the effects of entry regula- tions and firing costs on international income differences. Journal of Economic Growth, 17(2), 143-170. 190 Botero Juan, Simeon Djankov, Rafael Porta, Florencio C. Lopez-De-Silanes. (2004). The Regulation of Labor. The Quarterly Journal of Economics, MIT Press, vol. 119(4), pages 1339-1382 Branstetter, L., Lima, F., Taylor, L. J., and Venâncio, A. (2014). Do entry regulations deter entrepreneurship and job creation? Evidence from recent reforms in Por- tugal. The Economic Journal, 124(577), 805-832. Bricongne, J. C., Demertzis, M., Pontuch, P., and Turrini, A. (2016). Macroeconomic Relevance of Insolvency Frameworks in a High-debt context: an EU Perspec- tive (No. 032). Directorate General Economic and Financial Affairs (DG ECFIN), European Commission. Bruhn, M. (2008). License to sell: the effect of business registration reform on entre- preneurial activity in Mexico. The World Bank. Bruhn, M. and McKenzie, D. (2014). Entry regulation and the formalization of micro- enterprises in developing countries. The World Bank Research Observer, 29(2), 186-201. Calvino, F., C. Criscuolo and C. Menon (2016), “No Country for Young Firms?: Start- up Dynamics and National Policies”, OECD Science, Technology and Industry Policy Papers, No. 29, OECD Publishing, Paris, https://doi.org/10.1787/5jm22p- 40c8mw-en Campello, M. and Larrain, M. (2016). Enlarging the contracting space: Collateral Independent Evaluation Group World Bank Group    191 menus, access to credit, and economic activity. The Review of Financial Studies, 29(2), 349-383. Campos, F., Goldstein, M., and McKenzie, D. (2015). Short-term impacts of formal- ization assistance and a bank information session on business registration and access to finance in Malawi. The World Bank. Chari, A. V. (2011). Identifying the aggregate productivity effects of entry and size restrictions: An empirical analysis of license reform in India. American Eco- nomic Journal: Economic Policy, 3(2), 66-96. Chowdhury, F., Belitski, M., & Desai, S. (2014). Taxes, Corruption and Entrepreneur- ship. Academy of Management Proceedings (Vol. 2014, No. 1, p. 10559). Briarcliff Manor, NY 10510: Academy of Management. De Andrade, G. H., Bruhn, M., and McKenzie, D. (2013). A helping hand or the long arm of the law? Experimental evidence on what governments can do to formal- ize firms. The World Bank. De Mel, S., McKenzie, D. and Woodruff, C. (2013). The demand for, and consequences of, formalization among informal firms in Sri Lanka. American Economic Jour- nal: Applied Economics, 5(2), 122-50. Demirgüç-Kunt, A., Horváth, B. L., and Huizinga, H. (2017). How does long-term finance affect economic volatility? Journal of Financial Stability, 33, 41-59. Di Giovanni, J., & Levchenko, A. A. (2013). Firm entry, trade, and welfare in Zipf’s world. Journal of International Economics, 89(2), 283-296. The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix E Divanbeigi, R., and Ramalho, R. (2015). Business regulations and growth. The World Bank. Djankov, S., Freund, C., and Pham, C. S. (2010). Trading on time. The Review of Eco- nomics and Statistics, 92(1), 166-173. Djankov, S. and Ramalho, R. (2009). Employment laws in developing countries. Jour- nal of Comparative Economics, 37(1), 3-13. Doidge, C., Karolyi, G. A. and Stulz, R. M. (2007). Why do countries matter so much for corporate governance? Journal of financial economics, 86(1), 1-39. Fang, L. and Rogerson, R. (2011). Product market regulation and market work: a benchmark analysis. American Economic Journal: Macroeconomics, 3(2), 163- 88. Fernandes, N., Lel, U., and Miller, D. P. (2010). Escape from New York: The market impact of loosening disclosure requirements. Journal of Financial Economics, 95(2), 129-147. Fisman, R. and Svensson, J. (2007). Are corruption and taxation really harmful to growth? Firm level evidence. Journal of development economics, 83(1), 63-75. Freund, C. and Bolaky, B. (2008). Trade, regulations, and income. Journal of develop- ment economics, 87(2), 309-321. 192 Freund, C., Hallward-Driemeier, M., and Rijkers, B. (2014). Deals and Delays: Firm-Level Evidence on Corruption and Policy Implementation Times. World Bank Policy Research Working Paper No. 6949, Available at SSRN: https://ssrn. com/abstract=2460112 Geginat, C., and Ramalho, R. (2015). Electricity connections and firm performance in 183 countries. The World Bank. Haidar, J. I., Hoshi, T. and Shorenstein, W. H. (2015) Implementing Structural Re- forms in Abenomics: How to Reduce the cost of Doing Business In Japan. Hallward-Driemeier, M. and Pritchett, L. (2015). How business is done in the devel- oping world: Deals versus rules. Journal of Economic Perspectives, 29(3), 121-40. Haselmann, R., Pistor, K., and Vig, V. (2010). How law affects lending. The Review of Financial Studies, 23(2), 549-580. Hillberry, R. and Zhang, X. (2018). Policy and performance in customs: evaluating the trade facilitation agreement. Review of International Economics, 26(2), 438- 480. Houston, J. F., Lin, C., Lin, P., and Ma, Y. (2010). Creditor rights, information sharing, and bank risk taking. Journal of financial Economics, 96(3), 485-512. Ibrahim, M., and Alagidede, P. (2017). Financial Development, Growth Volatility and Information Asymmetry in Sub-Saharan Africa: Does Law Matter? South African Independent Evaluation Group World Bank Group    193 Journal of Economics, 85(4), 570-588. Iliev, Peter, Karl V. Lins, Darius P. Miller, and Lukas Roth. “Shareholder voting and corporate governance around the world.” The Review of Financial Studies 28, no. 8 (2015): 2167-2202. Iwanow, T., and Kirkpatrick, C. (2009). Trade facilitation and manufactured exports: Is Africa different?. World Development, 37(6), 1039-1050. Jales, Hugo. (2017). Estimating the Effects of the Minimum Wage in a Developing Country: A Discontinuity Design. Journal of Applied Econometrics 33: 29–51. Janiak, A. (2013). Structural unemployment and the costs of firm entry and exit. Labour Economics, 23, 1-19. Jerbashian, V. and Kochanova, A. (2016). The impact of doing business regulations on investments in ICT. Empirical Economics, 50(3), 991-1008. Kanbur, R. and Ronconi, L. (2016). Enforcement Matters: The Effective Regulation of Labor (No. 642-2016-44453). Kant, C. (2016). Are institutions in developing countries malleable? Journal of Policy Modeling, 38(2), 272-289. Kenny, C. (2007). Construction, corruption, and developing countries. The World Bank. Klapper, L. F., and Love, I. (2010). The impact of business environment reforms on The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix E new firm registration. World Bank policy research working paper, (5493). Klapper, L. F. and Love, I. (2010). The impact of business environment reforms on new firm registration. World Bank policy research working paper, (5493). Klapper, L., Laeven, L., and Rajan, R. (2006). Entry regulation as a barrier to entrepre- neurship. Journal of financial economics, 82(3), 591-629. Korinek, J. and P. Sourdin (2011). To What Extent Are High-Quality Logistics Services Trade Facilitating?. OECD Trade Policy Papers, No. 108, OECD Publishing, Paris, https://doi.org/10.1787/5kggdthrj1zn-en. Kovač, M. (2016). Institutional development, transaction costs and economic growth: evidence from a cross-country investigation. Journal of Institutional Econom- ics, 12(1), 129-159. Kraay, A. and Tawara, N. (2013). Can specific policy indicators identify reform priori- ties? Journal of Economic Growth, 18(3), 253-283. Lee, Y., and Gordon, R. H. (2005). Tax structure and economic growth. Journal of public economics, 89(5-6), 1027-1043. Li, W. (2019). Time barrier to export for OECD countries. Economics Letters, 175, 106-112. Martincus, C. V., Carballo, J., and Graziano, A. (2015). Customs. Journal of Interna- tional Economics, 96(1), 119-137. 194 Menezes, A. P. (2014). Debt resolution and business exit: insolvency reform for cred- it, entrepreneurship, and growth (No. 90759, pp. 1-8). The World Bank. Moïsé, E. and S. Sorescu (2013). Trade Facilitation Indicators: The Potential Impact of Trade Facilitation on Developing Countries’ Trade. OECD Trade Policy Papers, No. 144, OECD Publishing, Paris, https://doi.org/10.1787/5k4bw6k- g6ws2-en. Nangpiire, C., Rodrigues, R. G., and Adam, I. O. (2018). Ease of doing business and foreign direct investment inflow among Sub-Sahara African countries. Interna- tional Journal of Business and Emerging Markets, 10(3), 289-303. Nunn, N. (2007). Relationship-specificity, incomplete contracts, and the pattern of trade. The Quarterly Journal of Economics, 122(2), 569-600. Packard, Truman G., and Claudio E. Montenegro. (2017). Labor policy and digital technology use: indicative evidence from cross-country correlations. Policy Research Working Paper 8221, World Bank. Pinotti, P. (2012). Trust, regulation and market failures. Review of Economics and Statistics, 94(3), 650-658. Poel, K., Marneffe, W., Bielen, S., Van Aarle, B. and Vereeck, L. (2014). Administrative simplification and economic growth: A cross country empirical study. Journal of Business Administration Research, 3(1), 45 Independent Evaluation Group World Bank Group    195 Porta, R. L., & Shleifer, A. (2008). The unofficial economy and economic develop- ment (No. w14520). National Bureau of Economic Research. Portugal-Perez, A., and Wilson, J. S. (2009). Why trade facilitation matters to Afri- ca. World Trade Rev., 8, 379. Poschke, M. (2010). The regulation of entry and aggregate productivity. The Eco- nomic Journal, 120(549), 1175-1200. Schivardi, F. and Viviano, E. (2011). Entry barriers in retail trade. The Economic Jour- nal, 121(551), 145-170. Selwaness, Irène, and Chahir Zaki. (2015). On the Interaction between Trade Re- forms and Labor Market Regulation: Evidence from the MENA Countries’ Labor Markets. Working Paper, ERF. Shepherd, B. (2009). Speed money: time, corruption, and trade. MPRA Paper 17337, University Library of Munich, Germany. Van Stel, A., Storey, D. J. and Thurik, A. R. (2007). The effect of business regulations on nascent and young business entrepreneurship. Small business economics, 28(2), 171-186. Westmore, B. (2016). Scaling new heights: achievements and future challenges for productivity convergence in Lithuania. Williams, C. C. (2014). Tackling enterprises operating in the informal sector in devel- oping and transition economies: a critical evaluation of the neo-liberal policy approach. Journal of Global Entrepreneurship Research, 4(1), 1-17. The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix E 196 1  Klapper et al. (2006), Prantl (2010), Branstetter et al. (2013), Bruhn (2008), Divanbeigi and Ramalho (2015), Freund and Bolaky (2008). 2  Branstetter et al. (2013), Klapper and Love (2010), Ayyagari et al. (2007). 3  Formalization: Williams (2014), Campos et al (2015), Andrade et al. (2013), Bruhn and McK- enzie (2014), Mel et al. (2013), Pinotti (2012), Van Stel et al. (2007). Productivity: Boedo and Mukoyama (2012), Chari (2011) vs. Poschke (2010), Schivardi and Viviano (2011), Westmore (2016). Labor: Bruhn (2008), Schivardi and Viviano (2011), vs. Janiak (2013), Fang and Roger- son (2011). 4  Haselmann et al. (2010), Campello et al. (2016), Ibrahim et al. (2017). 5  Demirgüç-Kunt et al. (2017). 6  Houston F. et al. (2010). 7  Block (2016), Belitski et al. (2016), Andersson and Henrekson (2015), Lee and Gordon (2005), Becker et al. (2012), Lawless (2013), Fisman and Svensson (2007), Chowdhury et al (2014) vs. Di Giovanni, J., & Levchenko, A. A. (2013). 8  Block (2016), Block (2016), Baliamoune-Lutz and Garallo (2014), La Porta and Shleifer (2008), Jerbashian and Kochanova (2016). 9  Block (2016), Baliamoune-Lutz and Garallo (2014) 10  Nunn (2007). Independent Evaluation Group World Bank Group    197 11  Kovač (2016). 12  Demirgüç-Kunt, et al. (2017). 13  Ahlquist, and Prakash, A. (2010). 14  Sourdin and Korinek (2011), Iwanow and Kirkpatrick (2009), Moïsé and Sorescu (2013), Por- tugal-Perez and Wilson (2009), Djankov et al. (2010), Volpe Martincus et al. (2015). 15  Li (2019). 16  Hillberry, et al. (2018). 17  Portugal-Perez, A., et al. (2009). 18  Shepherd, B. (2009). 19  Poel, et al. (2014). 20  Freund, et al. (2014) and Kenny (2007). 21  Kraay, et al. (2013). 22  Hallward-Driemeier, and Pritchett (2015). 23  Freund, and Bolaky (2008). 24  Nangpiire, et al. (2018). 25  Hallward-Driemeier, and Pritchett (2015). 26  Doidge, et al. (2007). The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix E 27  Fernandes et al. (2010), Athari et al. (2016) and Iliev, et al. (2015) 28  Geginat and Ramalho (2018). 29  Ibid. 30  Ibid. 31  Haidar I., Hoshi and Shorenstein (2015). 32  ibid. 33  Arlet (2017) 34  Calvino et al. (2016). 35  Becker, B., and Josephson (2016). 36  Demirgüç-Kunt et al. (2017). 37  Menezes (2014). 38  Bricongne C., Demertzis, Pontuch and Turrini (2016). 39  ibid 40  Djankov and Ramalho (2009), Jales (2017), Acar et al. (2019). 41  Selwaness and Zaki (2015). ibid. 198 42  43  Packard et al. (2017). 44  Kanbur, et al. (2016). 45  Botero J., et al. (2004). 46  Truman, and Montenegro (2017). 47  Acharya et al. (2013). Independent Evaluation Group World Bank Group    199 Appendix F. Structured Literature Review (SLR) Introduction: The DB project and its rankings are now widely credited with influencing business regulations worldwide, dominating market share among business climate indicators and motivating many regulatory changes as countries vie for improved status in its rankings (Doshi et al. 2019; Besley 2015). Is there evidence that the types of regulatory reforms promoted by the DB project have an impact on economic and development The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix F outcomes? This has become a particularly salient issue in recent years given many experts’ scepticism of DB’s narrative that reforms targeting the factors captured by the indicators improve economic development and growth. Some studies show that the correlations between the DB indicators and development outcome variables are unstable (Kraay and Tawara, 2013) or suffer from omitted variable bias (Pinotti, 2008). These issues begin to shed light on some of the broader attribution problems associated with conclud- ing the effects of reforms from correlations between outcome variables and the DB indicators. Ccorrelation alone is not sufficient to assess the likely consequence of regulatory reform inspired by or related to the indicators (Branstetter et al. 2014; Altenburg et al., 2017). Endogeneity and measure- ment issues limit the causal claims associated with such correlations (de Mel et al., 2013). Furthermore, parts of the indicators can be determined by changes in external contextual factors that may have nothing to do with the regulatory process, so it is not always possible to attribute changes in the index solely to regulatory adjustments or the outcomes of reforms promoted by DB (Chemin, 2009 and Branstetter et al. 2014). Overall, while DB indica- tors enable countries to compare regulatory environments and may also help to identify contexts where businesses experience a more onerous regulatory process, they do not provide precise evidence of the effects of reforms seek- ing to alleviate these issues. Approach: This review aims to draw together more direct and rigorous ev- idence of the effects of changes in regulatory arrangements. These studies 200 exploit natural experiments or variations in the implementation of reforms (such as the variation in the time or region of the reform) to evaluate their effects. Prior reviews often only draw on a selective range of the literature or focus on small sections of DB’s total regulatory portfolio. This review seeks to provide an updated view of this literature, taking stock of the evidence available on the effects of business regulatory reforms in each of the DB regulatory categories. By identifying the current state of knowledge, this SLR seeks to help to crit- ically appraise the relevance of theories, to provide a basis for interventions, to guide future studies and to summarize information and understanding in 10 areas of DB coverage. A systematic [structured] literature review aims to minimize researcher bias by collating and appraising all available research that has been identified using an explicit literature search strategy and meets pre-specified eligibility criteria (Higgins et al, 2019). IEG’s structured literature review approach reflects a rapid evidence review guided by the concept of a systematic review but tailored to meet the constraints of a typi- cal WBG evaluation project. Summary of inclusion and exclusion criteria. This structured literature review adheres to pre-specified criteria defining the characteristics of an includible study, the method of data (information) extraction and synthesis and reports a specific search strategy. The study inclusion criteria follow the PICOS(LY) approach common to many systematic reviews [Table 1] (Littell Independent Evaluation Group World Bank Group    201 and White, 2017). The PICOS(LY) approach specifies the characteristics of eligible studies; outlining includable populations, interventions, comparison groups (effectiveness studies only), outcomes, study designs, the written lan- guage of the study, and year of study (see more below). Reflecting its short timeframes and resource constraints, both the search strategy and analytic methods applied in this SLR were more limited than in a full systematic literature review. nclusion and exclusion criteria for empirical studies: PICOS(LY) Table F.1. I summary table. Criteria Description Population No limit – studies may focus on population groups in any country. Intervention See intervention framework in Appendix 1. This review includes studies examining regulatory reforms and interventions related to the Doing Business project based on a review of the Doing Business Reform Data- base. It excludes empirical studies which do not examine the effects of a specific intervention or reform1. Comparison Control groups may be subject to no intervention, on a waitlist, or part of an alternative intervention or condition. Outcome The review does not exclude based on the outcome of a study, except The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix F when considering related infrastructure projects. Studies examining the impact of infrastructure projects (e.g., in the getting electricity domain) are limited to business outcomes, recognizing the broader societal effects they may also have which may not necessarily be attributable to changes in the business environment. Study Design Includes studies where allocation to intervention and control groups are random or selection bias has been addressed by design. It excludes sim- ple before-and-after comparisons and simulation and forecast models. Language Includes studies written in English only. Year No limit – but the latest search date was February 2021. Source: Independent Evaluation Group. The DB reform framework used does not capture all possible types of inter- ventions that may be being implemented in each regulatory category. In- terventions outside of the immediate focus of the DB project (as identified in the DB reform database) are omitted from the review. The SLR also does not capture all interventions within DB topic areas that do not meet the assumptions of its stylized business case model (such as reforms to special investment or tax regimes or export processing zones). The review includes studies no matter what populations are examined -- (e.g., high-, middle-, or low-income economies) and whether or not it has been published in an academic peer-reviewed journal or exists as a working paper or institution- al report. It includes all years of study until Feb. 2021 (the date of the final search). In other words, it looks at historic evidence of the effects of relevant regulatory reforms, as well as more recent evaluations. It screens only arti- cles written in English. 202 The SLR includes quantitative studies addressing reforms effects by applying one of the following approaches: a. Allocation of participants to intervention and control groups using a ran- domized or quasi-randomized mechanism at individual or cluster levels. b. Non-randomized designs with selection on unobservables. This includes natural experiments using a sharp or fuzzy regression discontinuity de- sign, studies using panel or pseudo-panel data with estimation strategies to account for time-invariant unobservables (e.g. fixed-effects models), and cross-sectional studies using multi-stage or multivariate approaches to account for unobservables (inc. instrumental variable and Heckman two-step type estimation approaches). c. Non-randomized designs with selection on observables. This includes cross-sectional or panel (i.e. controlled before and after) studies that use a method to statistically match or weight observations in the intervention and comparison groups. For example, includes studies using a form of propensity score matching or entropy balancing. Our definition for the comparison condition of a control group is not strict and may comprise of observations subject to no intervention, on a wait- list, or a member of an alternative intervention or condition. Excluded are studies that use simulation or forecast models and case studies that do not satisfy the methodological conditions described above. The methodologi- Independent Evaluation Group World Bank Group    203 cal criteria described here is consistent with norms established by leading practitioners of systematic reviews, including the International Initiative for Impact Evaluation (3ie) and the Campbell Collaboration. Search strategy: Identifying a comprehensive list of studies for this review is challenging because of the scope and breadth of the various aspects of the regulatory environment covered by the DB project. Taking into con- sideration the review’s resource constraint and to maximize the likelihood of finding relevant studies, the search started by screening two databases with a high probability of providing relevant studies. The first was the DB team’s own database on studies related to the DB project. At the time of the search, December 2020 to February 2021, this database provided a selection of 426 topic-specific research papers. The included papers had a strong bias towards studies from outlets ranked in the top 100 economic journals and working papers by RePEc (Research Papers in Economics) and consisted of a broad range of literature reviews, econometric studies, and case studies related to the topics covered by the doing business indicators. The second database searched is the online evidence portal provided by 3ie, a database of more than 3,500 research papers identified through a long-term project searching for studies meeting the study design criteria discussed above. 3ie’s project has screened more than 150,000 records from academic databases in health, economics, public policy and the social sciences listed on platforms such as Ovid, EbscoHost and ProQuest. It has also searched library databases and websites from select research organizations, repositories, and academic The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix F institutions (such as the World Bank, Inter-American Development Bank, African Development Bank, Social Science Research Network, IDEAS, etc.). Studies from sectors that are not relevant (e.g. education, health, etc.) were omitted, leaving 1206 records from relevant sectors (e.g. industry trade and services and public administration) to examine (figure 1). To supplement the search for relevant literature the references listed were of the included papers (i.e. conducted backwards citation tracking) and a selection of rele- vant literature reviews on Doing Business and its related reforms. This was also complemented with citation tracking using Google Scholar’s electronic citation tracking system to identify studies referencing each of these studies (i.e. forward citation tracking). In total, after removing duplicate records, 9,221 records were screened for rel- evance based on their title and abstract and 1,894 records at full text (see Fig- ure 1). Screening at title and abstract removed studies with no clear relevance to the topic being reviewed (e.g. studies not related to business regulation or the reforms identified in appendix 1). Full-text screening removed studies that did not specifically meet the inclusion criteria outlined in section 2.1 (e.g., because they did not meet the study design criteria). Of the 1,894 manuscripts screened at full-text, 103 records met the inclusion criteria. Data Extraction and Synthesis: Data was extracted from each record included in the review. The data consists of bibliographic information (such as author names, study title, year of publication, and publications status) and study characteristics. The details of the study characteristics coded include infor- 204 mation on their main findings, type of intervention or regulatory reform, geographic focus and method of analysis.  istribution of Evidence and Findings of Positive Effects, by Figure F.1. D DB Area Source: Independent Evaluation Group. Summary of findings: This review finds a relatively rich and growing body of evidence on the effects of business regulatory reforms as a whole, including at least 103 studies (see reference list). Figure F1 depicts the highly uneven distribution of identified studies across the Doing Business topics. Evidence is concentrated in a few areas, such as starting a business, getting credit, and protecting minority investors. Evidence gaps, where the reforms have little Independent Evaluation Group World Bank Group    205 or no rigorous evidence, appear on the topics of registering property and getting electricity. Of the evidence available, several evaluations report in- stances where reforms making starting a business simpler, cheaper, and less demanding in its requirements have helped to promote firm formalization, increase growth, and improve competition. Similarly, studies also point to the important contributions of reforms im- proving credit information and collateral laws on firms access to finance, the trade enhancing effects of reforms promoting expedited trade regimes, and the value of tax administration reforms to governments efforts to improve the rate or efficiency of tax collection.  RISMA diagram (summary of the flow of records in the re- Figure F.2. P view) The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix F Source: Independent Evaluation Group. Figure F.2 shows a large proportion of the total number of studies (88 of 103 studies or 85%) provide evidence of the positive outcomes of reforms. Over- all, this evidence emphasizes both the relevance and potential significance of the DB project. However, the evidence identified also highlights there is currently an im- balance in the treatment of theory and evidence reported by the DB reports. The reports mainly cite studies favoring the reforms they promote; creating an advocacy tool rather than providing an adequate piece of research or an evaluation of the issues they are concerned with. The project requires more transparent and systematic reporting of the available evidence, with more attention to nuance and complexity in this literature. The following three points currently receive almost no attention in DB’s own literature reviews: 1. The indicators imply reforms that improve the cost and speed of factors are always beneficial, but this neglects that many regulations are also intended to serve socially valuable purposes (e.g. enhancing public health, 206 safety, the environment and reducing corruption). The findings for the En- forcing Contracts topic provide an interesting example; highlighting that in some instances judicial reforms may be balancing a trade-off between judicial efficiency and the quality of judicial decisions (Kondylis and Stein, 2015). Whether the DB indicators motivate countries to create reforms that prioritize the factors captured by the indicator to an extent that may also be harmful to other aspects of public policy is an important point that requires further research. 2. Despite the positive outcomes reported in many evaluations and empha- sized in the DB annual reports, the literature also highlights the possi- bility of several unintended (adverse) effects of these reforms. 14 of the 103 studies identified highlighted a potentially adverse effect of reforms. For example, the findings for the Starting a Business topic points to ex- amples of the possible negative consequences of reforms on factors such as the gender pay gap and environmental outcomes. Evidence is limited on such issues and further research is required to confirm and replicate these findings. 3. Cases of flagship interventions with evaluations reporting encouraging outcomes do not mean that all similar reforms in other countries or con- texts will have the same or similar effects. Greater consideration of the external validity of findings is required before generalizing individual findings to a broad range of contexts. In some instances, DB regulatory Independent Evaluation Group World Bank Group    207 reforms may also need to be coupled with changes in additional factors not covered by the indicators to make a meaningful or material differ- ence. For example, the evidence identified for the Starting a Business topic discusses problems related to entrepreneurs’ land tenancy rights undermining efforts to encourage formality (de Mel et al., 2013). DB provides little to no insight into how countries can identify or integrate such broader reform priorities. Box F.1. Limitations 1. This review’s literature search has screened many more studies than most full systematic reviews, yet the breadth of the potentially relevant literature means that inevitably some unidentified evidence may still exist. Future research should focus on continuing to record and expand on the available evidence on the effects of reforms. The rich database of studies provided by this review offers a valuable resource for future work to start from. 2. While this analysis focused on recording studies’ main findings using a descriptive synthesis, another approach would extend to reporting and aggregating estimates of the effects of reforms in a statistical meta-analysis. However, any extensions should The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix F also consider that a meta-analysis only really provides meaningful results if the under- lying studies being aggregated represent something that are comparable or the same. (Duvendack et al., 2012). The DB reform portfolio covers many different types of reforms targeting very different types of intermediate outcomes introduced under very different conditions. Lumping together very different types of reforms or outcomes to approxi- mate some average effect of reforms may ignore important degrees of heterogeneity. 3. An extension of this synthesis could comprise systematic appraisals of the includ- ed studies’ risk of bias. This is to assert the confidence we might place in a particular study’s findings (see Moyer and Finney, 2005; Sterne et al., 2016; Fenton Villar and Waddington, 2019). Appraisals would be ideally completed with two independent as- sessors and a trained expert adjudicator. Currently, very little effort has been made to thoroughly appraise this evidence base. Further research would be needed to rigor- ously appraise the available evidence while also confirming the original assessments with the most recent line of expert appraisal guidelines. Source: Independent Evaluation Group. Finally, evidence of the positive impacts of reforms does not say much about how well the indicators measure or represent the issues these reforms are trying to address. The review raises questions about how well the Construc- tion Permits indicator (based on the construction of a warehouse) represents other aspects of planning regulation, or if it is representative of the require- ments of modern businesses across different types of economies and indus- 208 tries (or the growing service economy). Other examples concern the method used to calculate the time taken to complete procedures and whether this reflects the amount of time taken in reality (e.g. see Cappiello, 2014). The aggregation and implicit weighting scheme the indicators apply will inevita- bly attract attention too.2 Conclusion: This structured literature review screened 9,221 studies and identified 103 evaluations of relevant reforms where allocation to intervention and control groups are random or selection bias has been addressed by design (including natural experiments and non-randomized designs). These findings point to a rich and growing literature on the effects of business regulatory reforms relevant to the DB project. However, the identified evidence is dis- tributed highly unevenly across DB topic areas. This review also highlights the imbalance in the treatment of theory and evidence reported by the DB reports (mainly citing studies favoring the reforms they promote) and the need for more transparent and systematic reporting of the available literature. The existing DB literature pays insufficient attention to the complexity of reforms and their relationships with outcomes. A more nuanced understanding is required of the potential trade-offs reforms make between public policy goals, the context of the reform and the generalizability of the findings from individ- ual studies, and the risk of adverse effects arising from reforms. Independent Evaluation Group World Bank Group    209  ummary of Rigorous Findings from Structured Literature Table F.2. S Review Area What is Measured Positive Outcomes Cautions Starting a Procedures, time, cost, » The evidence high- » The literature Business and paid-in minimum lights some of the indicates these capital to start a lim- benefits associated reforms may ited liability company with reforms making also have sev- for men and women starting a business eral relatively simpler, cheaper, unexplored and the require- unintended ments less demand- consequences ing including: they (e.g. increasing The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix F can help to promote the gender firm formalization, pay gap and increase growth, and environmental improve competition. pollution). Construction Procedures, time, and » The broader liter- » Limited evidence Permits cost to complete all ature on planning exists directly formalities to build a regulation reforms verifying the ef- warehouse and the supports that less fects of reforms quality control and stringent planning changing con- safety mechanisms regulation can struction permit in the construction support both local regulations for permitting system economic outcomes warehouse con- (such as employ- struction. ment) and market » It is unclear competition. how well the construction of a warehouse represents other aspects of plan- ning regulation (e.g. concerning retail and office buildings). (continued) 210 Area What is Measured Positive Outcomes Cautions Getting Elec- Procedures, time, and » Several studies » This review did tricity cost to get connected emphasize the not identify any to the electrical grid; importance of the evidence of the reliability of the indicators underlying the effects of electricity supply; and construct concerning reforms on this the transparency of access to a reliable topic. The pauci- tariffs electricity supply ty of identifiable to businesses. The evidence in the ease of getting major publica- electricity rankings tions, literature also appear to be reviews, and positively correlated the database on with the perceptions Doing Business of the quality of the (which has also electricity supply. been supple- mented by the 3ie database) highlights this area warrants further research. Registering a Procedures, time, and » The limited evidence » The evidence Property cost to transfer a prop- available indicates on the effects erty and the quality of reforms intending to of reforms is the land administration improve the accessi- relatively sparse system for men and bility of land services and very little women and the process evidence has determining the been identified transfer of property on the effects may help increase of reforms on activity on commer- broader out- Independent Evaluation Group World Bank Group    211 cial rental and prop- comes (such as erty markets, as well output or firm as access to credit. growth). (continued) Area What is Measured Positive Outcomes Cautions Getting Movable collateral » The evidence is » Credit informa- Credit laws and credit infor- mixed but generally tion reforms mation systems points to the credit may benefit enhancing effects of high-quality reforms expanding borrowers while credit bureaus, reg- low-quality istries, and the menu borrowers (ones of moveable assets with a weaker that may be used as credit history) collateral. may find credit » Evidence of the harder to come expansion of credit by. This raises information systems the question whether these The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix F particularly high- lights their benefits information of improving firms’ reforms stifle fi- repayments and de- nancial inclusion linquency, although in favour of im- their effects only proved terms for seem to appear with already favored time, as the system borrowers. expands. » More expan- sive collateral laws seem to increase credit to otherwise financially constrained businesses, but a trade-off exists that these regulations do not promote over-indebt- edness (risking greater volatility and financial distress). (continued) 212 Area What is Measured Positive Outcomes Cautions Protecting Minority shareholders’ » Reforms strengthen- » Reforms eco- minority rights in related-party ing investor protec- nomic effects investors transactions and in tion have several are ambiguous; corporate governance potential economic in some con- benefits (inc. increas- texts, they may ing firm value and increase the shareholder divi- cost of equity, dends and curbing debt, and audit exploitive business fees. power and tunneling » Evidence on activities). DB’s array of ‘policy prescrip- tions’ informing the indicator is limited. Paying Taxes Payments, time, and » Tax administration » Market and insti- total tax and contri- reforms increased tutional factors bution rate for a firm tax payments and may be import- to comply with all tax helped to reduced ant determi- regulations as well as tax evasion. nants of reforms post-filing processes » Some evidence also outcomes (e.g. exists they improve small effects, if the perception of tax any, in sectors administration as an with low incen- obstacle to firms’ op- tives to formal- eration and growth. ize and where tax compliance is already high). Trading Time and cost to » Evidence points to » Insufficient at- Independent Evaluation Group World Bank Group    213 across bor- export the product of the trade enhancing tention has been ders comparative advan- effects of reforms given to the tage and to import a promoting expedited implications of standardized cargo of trade regimes. reforms on other goods by sea public objec- tives (e.g. public health, safety, the environment and reducing informality). (continued) Area What is Measured Positive Outcomes Cautions Enforcing Time and cost to » Evaluations of the » The perfor- contracts resolve a commercial effects of reforms mance mea- dispute and the quality to improve contract sures currently of judicial processes enforcement point to captured by the for men and women the positive effects DB indicators do they can have on not account for improving judicial the potential ef- efficiency and firms ficiency-quality experience with the trade-off caused judicial process. by judicial re- » Evidence indicates forms. this may impact » Evidence on the several broader out- various ‘good The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix F comes, such as firm practices’ pro- investment, produc- moted by the tivity, and returns. DB indicator is limited. (continued) 214 Area What is Measured Positive Outcomes Cautions Resolving Time, cost, outcome, » Evidence on the ef- » The literature insolvency and recovery rate for fects of reforms that also highlights commercial insolvency improve the efficien- the trade- and the strength of the cy of the insolvency off between legal framework for process indicates increasing cred- insolvency that they may have a itors strength in significant effect on the insolvency the cost of borrow- process and ing and firms access deterring firms to finance. from borrow- ing. Currently, the weighting scheme applied within the DB indicator means that it places greater empha- sis on reforms that improve creditors protec- tion. » Some evidence exists showing reforms out- comes may be subject to a substitution effect between firms’ labor and Independent Evaluation Group World Bank Group    215 capital in- vestment. It is unclear whether in some con- texts the labor market effects of these reforms can contradict social policy objectives (such as increasing incomes or alle- viating poverty). Source: Independent Evaluation Group. References Aboal, D., Noya, N. and Rius, A. 2014. Contract enforcement and investment: A sys- tematic review of the evidence. World Development, 64: 322-338. Aghion, P., Burgess, R., Redding S. and Zilibotti, F. 2008. The unequal effects of liber- alization: Evidence from dismantling the license raj in India. American Economic Review, 98(4): 1397-1412. Aghion, P., Burgess, R., Redding S. and Zilibotti, F. 2005. Entry Liberalization and Inequality. Journal of the European Economic Association, 3(2): 291-302. Ali, D., Deininger, K. and Duponchel, M. 2016. Using Administrative Data to Assess the The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix F Impact and Sustainability of Rwanda’s Land Tenure Regularization. World Bank Policy Research Working Paper, No. 7705. Alok, S., Chaurey, R. and Nukala,V. 2017. Creditor Rights and Corporate Labor Policy: Evidence from a Policy Experiment. Working Paper. Altenburg, T., Hampel-Milagrosa, A. and Loewe. 2017. A Decade On: How Relevant is the Regulatory Environment for Micro and Small Enterprise Upgrading After All. The European Journal of Development Research, 29; 457-475. Amici, M., Giacomelli, S. Manaresi, F. and Tonello, M. 2016. Red tape reduction and firm entry: New evidence from an Italian reform. Economics Letters, 146: 24-27. Aparicio, G. 2014. Does Formality Improve Firm Performance? Evidence From a Qua- si-Experiment in Mexico. Working Paper. Araujo, A. Ferreira, R. and Funchal, B. 2012. The Brazilian bankruptcy law experi- ence. Journal of Corporate Finance, 18(4): 994-1004. Aretz, K., Campello, M. and Marchica, M. 2020. Access to Collateral and the Democ- ratization of Credit: France’s Reform of the Napoleonic Security Code. Journal of Finance, 75(1): 45-90. Armour, J., Menezes, A., Uttamchandani, M. and van Zwieten, K. 2015. How do credi- tor rights matter for debt finance? A review of empirical evidence. Working Paper. Ayyagari, M., Beck, T. and Demirguc-Kunt, A. 2007. Small and medium enterprises across the globe. Small Business Economics, 29: 415-434. 216 Ayyagari, M., Juarros, P., Martinez Peria, M. and Singh, S. 2016. Access to Finance and Job Growth: Firm-Level Evidence across Developing Countries. World Bank Policy Research Working Paper, No. 7604. Bae, K., El Ghoul, S., Guedhami, O. and Zheng, X. 2020. Board Reforms and Dividend Policy: International Evidence. Journal of Financial and Quantitative Analysis, 56(4): 1296-1320. Balakrishnan, K. and A. Ertan 2019. Credit Information Sharing and Loan Loss Recog- nition. Working Paper. Beg, S. 2019. Computerization and Development: Formalizing Property Rights and its Impact on Land and Labor Allocation. Working Paper. Behr, P. and S. Sonnekalb. 2012. The effect of information sharing between lenders on access to credit, cost of credit, and loan performance - Evidence from a credit registry introduction. Journal of Banking and Finance, 36(11): 3017-3032. Bellon, M., Chang, J., Dabla-Norris, E., Khalid, S., Lima, F., Rojas, E. and Villena, P. 2019. Digitalization to Improve Tax Compliance: Evidence from VAT e-Invoicing in Peru. IMF Working Paper, No. 19/231. Bertrand, M. and Kramarz, F. 2002. Does entry regulation hinder job creation? Evidence from the French retail industry. The Quarterly Journal of Economics, 117(4): 1369-1413. Independent Evaluation Group World Bank Group    217 Besley, T. 2015. Law, regulation, and the business climate: The nature and influence of the World Bank Doing Business project. Journal of Economic Perspectives, 29(3): 99-120. Black, B. and Kim, W. 2012. The effect of board structure on firm value: A multiple identification strategies approach using Korean data. Journal of Financial Eco- nomics, 104(1): 203-226. Black, B., Kim, W., Jang, H. and Park, K. 2015. How corporate governance affect firm value? Evidence on a self-dealing channel from a natural experiment in Korea. Journal of Banking and Finance, 51: 131-150. Black, B., Kim, W. and Nasev, J. 2019. The Effect of Board Structure on Firm Disclo- sure and Behavior: A Case Study of Korea and a Comparison of Research De- signs. Working Paper. Blanco-Perez, C. and Brodeur, A. 2020. Publication Bias and Editorial Statement on Negative Findings. The Economic Journal, 130(629): 1226-1247. Branstetter, L., Lima, F., Taylor, L. and Venancio, A. 2014. Do entry regulations deter entrepreneurship and job creation? Evidence from recent reforms in Portugal. The Economic Journal, 124(577): 805-832. Braun, R., Eidenmuller, H., Engert, A. and Hornuf, L. 2013. Does Charter Competition Foster Entrepreneurship? A Difference-in-Difference Approach to European The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix F Company Law Reforms. Journal of Common Market Studies, 51(3): 399-415. Bruhn, M. 2011. License to sell: The effect of business registration reform. The Re- view of Economics and Statistics, 93(1): 382-386. Bruhn, M. 2013. A tale of two species: Revisiting the effect of registration reform on informal business owners in Mexico. Journal of Development Economics, 103: 275-283. Bruhn, M., Cho, C., Marusic, A., Nguyen, H., Reyes, J. and Tran, T. 2018. Courts and Business Registration: Evidence from Serbia. Courts and Business Registration: Evidence from Serbia. World Bank Policy Research Working Paper, No. 8611. Bruhn, M. and McKenzie, D. 2013. Using administrative data to evaluate municipal reforms: an evaluation of the impact of Minas Fácil Expresso. Journal of Develop- ment Effectiveness, 5(3): 319-338. Bruhn, M. and McKenzie, D. 2014. Entry Regulation and the Formalization of Micro- enterprises in Developing Countries. The World Bank Research Observer, 29(2): 186-201. Büyükkarabacak, B. and Valev, N. 2012. Credit information sharing and banking cri- ses: An empirical investigation. Journal of Macroeconomics, 34(3): 788-800. Cadot, O., Fernandes, A., Gourdon, J. and Mattoo, A. 2011. Where to Spend the Next Million? Applying Impact Evaluation to Trade Assistance. Washington, DC, The International Bank for Reconstruction and Development/The World Bank. 218 Cameron, D. Mishra, A. and Brown, A. The growth of impact evaluation for interna- tional development: how much have we learned. Journal of Development Effec- tiveness, 8(1); 1-21. Campello, M. and Larrain, M. 2015. Enlarging the contracting space: Collateral menus, access to credit, and economic activity. Review of Financial Studies, 29(2): 349-383. Canare, T., Ang, A. and Mendoza, R. 2016. Ease of Doing Business: International Policy Experience and Evidence. Working Paper. Cappiello, A. 2014. World Bank Doing Business Project and the statistical methods based on ranks: the paradox of the time indicator. Rivista Italiana di Economia, Demografia e Statistica, 68(1): 79-86. Carballo, J., Graziano, A., Schaur, G. and Vlope Martincus, C. 2016. The Border La- byrinth: Information Technologies and Trade in the Presence of Multiple Agencies. IDB Working Paper, No. 706. Carballo, J., Graziano, A., Schaur, G. and Vlope Martincus, C. 2016b. The Border Lab- yrinth: Information Technologies and Trade in the Presence of Multiple Agencies. IDB Working Paper, No. 706. Carballo, J., Schaur, G., Graziano, A., and Volpe Martincus, C. 2016a. Transit Trade. IDB Working Paper, No. 704. Independent Evaluation Group World Bank Group    219 Carballo, J., Schaur, G., Graziano, A., and Volpe Martincus, 2016c. Posts as Trade Fa- cilitators Posts as Trade Facilitators. IDB Working Paper, No. 701. Carballo, J., Schaur, G., Graziano, A., and Volpe Martincus. 2016d. Trust no one? Secu- rity and International Trade. IDB Working Paper, No. 703. Castellaneta, F., Conti, R. and Kacperczyk, A. 2020. The (Un) intended consequences of institutions lowering barriers to entrepreneurship: The impact on female workers. Strategic Management Journal, 41(7): 1274-1304. Chamarbagwala, R. and Sharma, G. 2011. Industrial de-licensing, trade liberalization, and skill upgrading in India. Journal of Development Economics, 96(2): 314-336. Chari, A. 2011. Identifying the aggregate productivity effects of entry and size re- strictions: An empirical analysis of license reform in India. American Economic Journal: Economic Policy, 3(2): 66-96. Chemin, M. 2009. The impact of the judiciary on entrepreneurship: Evaluation of Pakistan’s “Access to Justice Programme”. Journal of Public Economics, 93(1-2): 114-125. Chemin, M. 2012. Does court speed shape economic activity? Evidence from a court reform in India. Journal of Law, Economics, and Organization, 28(3): 460-485. Chemin, M. 2020. Judicial efficiency and firm productivity: Evidence from a world database of judicial reforms. Review of Economics and Statistics, 102(1): 49-64. The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix F Chemin, M., Harley, G., Panter, E. and Rene, E. 2017. What Works? Examples of Empir- ically Proven Justice Reforms. World Bank Group. Chen, R., Guedhami, O., Yang, Y. and Zaynutdinova, G. 2020. Corporate governance and cash holdings: Evidence from worldwide board reforms. Journal of Corporate Finance, 65: 101771. Chen, Y. and Du, J. 2020. Does regulatory reform of cumulative voting promote a more balanced power distribution in the boardroom. Journal of Corporate Fi- nance, 64; 101655. Cheng, X. and Degryse, H. 2010. Information Sharing and Credit Rationing: Evidence from the Introduction of a Public Credit Registry. European Banking Center Dis- cussion Paper, No. 07S. Choudhary, M. A. and A. K. Jain (2020). How public information affects asymmetri- cally informed lenders: Evidence from a credit registry reform. Journal of Devel- opment Economics, 143: 102407. Ciccone, A. and Papaioannou, E. 2007. Red tape and delayed entry. Journal of the European Economic Association, 5(2-3): 444-458. Cirmizi, E., Klapper, L. and Uttamchandani, M. 2012. The challenges of bankruptcy reform. World Bank Research Observer, 27(2): 185-203. Closset, F. and Urban, D. 2019. The balance of power between creditors and the firm: Evidence from German insolvency law. Journal of Corporate Finance, 58: 454-477. 220 Courseuil, C. and Moura, R. 2010. The Effects Of Tax Incentives For Small Firms On Employment Levels. Working Paper. Dammann, J. 2017. Business Courts and Firm Performance. Working Paper. de Andrade, G., Bruhn, M. and McKenzie, D. 2016. A helping hand or the long arm of the law? Experimental evidence on what governments can do to formalize firms. World Bank Economic Review, 30(1): 24-54. De Haas, R. and Millone, M. 2020. The Impact of Information Sharing on the Use of Collateral versus Guarantees. World Bank Economic Review, 34: S14-S19. de Janvry, A., McIntosh, C., and Sadoulet, E. 2010. The supply- and demand-side impacts of credit market information. Journal of Development Economics, 93(2): 173-188. de Mel, S., McKenzie, D. and Woodruff, C. 2013. The demand for, and consequences of, formalization among informal firms in Sri Lanka. American Economic Journal: Applied Economics, 5(2): 122-150. Deininger, K. and A. Goyal 2012. Going digital: Credit effects of land registry com- puterization in India. Journal of Development Economics, 99(2): 236-243. Demenet, A., Razafindrakoto, M. and Roubaud, F. 2016. World Development, 84; 326- 341. Demirgüç-Kunt, A., Horvath, B. and Huizinga, H. 2017. How does long-term finance Independent Evaluation Group World Bank Group    221 affect economic volatility? Journal of Financial Stability, 33: 41-59. Ding, W., Wei, L. and Xie, W. 2020. Does Bank Credit Affect Innovation? Evidence from Collateral Law Reform. Working Paper. Djankov, S. 2009. The regulation of entry: A survey. World Bank Research Observer, 24(2): 183-203. Djankov, S., La Porta, R., Lopez-de-Silanes, F. and Shleifer, A. 2002. The Regulation of Entry. Quarterly Journal of Economics, 117(1): 1-37. Djankov, S., McLiesh, C. and Ramalho, R. 2006. Regulation and growth. Economics Letters, 92(3): 395-401. Doblas-Madrid, A. and Minetti, R. 2013. Sharing information in the credit market: Contract-level evidence from U.S. firms. Journal of Financial Economics, 109(1): 198-223. Doshi, R., Kelley, J. and Simmons, B. 2019. The Power of Ranking: The Ease of Doing Business Indicator and Global Regulatory Behavior. International Organization, 73(3): 611-643. Dreher, A. and Gassebner, M. 2013. Greasing the wheels? The impact of regulations and corruption on firm entry. Public Choice, 155: 413-432. Duvendack, M. and Mader, P. Impact of financial inclusion in low- and middle-in- come countries: A systematic review of reviews. Campbell Systematic Reviews, The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix F 15(1-2): e1012. Duvendack, M., Hombrados J., Palmer-Jones, R. and Waddington, H. Assessing ‘what works’ in international development: meta-analysis for sophisticated dummies. Journal of Development Effectiveness, 4(3); 456-471. Evans, D. and Popova, A. 2016. What Really Works to Improve Learning in Develop- ing Countries? An Analysis of Divergent Findings in Systematic Reviews. The World Bank Research Observer, 31(2): 242-270. Fajnzylber, P., Maloney, W. and Montes-Rojas, G. 2011. Does formality improve mi- cro-firm performance? Evidence from the Brazilian SIMPLES program. Journal of Development Economics, 94: 262-276. Fan, Y., Jiang, Y., Kao, M. and Liu, F. 2020. Board independence and firm value: A quasi-natural experiment using Taiwanese data. Journal of Empirical Finance, 57: 71-88. Fauver, L., Huang, M., Li, X. and Taboada, A. 2017. Board reforms and firm value: Worldwide evidence. Journal of Financial Economics, 125(1): 120-142. Fenton Villar, P. and Waddington, H. 2019. Within study comparisons and risk of bias in international development: Systematic review and critical appraisal. Camp- bell Systematic Reviews, 15(1-2): e1027. Fernandes, A., Ferreira, P. and Winters, L. 2014. Firm entry deregulation, competition and returns to education and skill. European Economic Review, 70: 210-230. 222 Fernandes, A., Hillberry, R. and Alcantara, A. 2021. Trade Effects of Customs Reform: Evidence from Albania. The World Bank Economic Review, 35(1): 34-57. Fernandes, A., Hillberry, R. and Berg, C. 2016. Expediting Trade: Impact Evaluation of an In-House Clearance Program. World Bank Policy Research Working Paper, No. 7708. Floridi, A., Demena, B. and Wagner, N. 2020. Shedding light on the shadows of in- formality: A meta-analysis of formalization interventions targeted at informal firms. Labour Economics, 67: 101925. Floridi, A., Demena, B. and Wagner, N. 2021. The bright side of formalization poli- cies! Meta-analysis of the benefits of policy-induced versus self-induced for- malization. Applied Economics Letters, 1-6. Freund, C. and Bolaky, B. 2008. Trade, regulations, and income. Journal of Develop- ment Economics, 87(2): 309-321. Fried, J., Kamar, E., Yafeh, Y. 2020. The effect of minority veto rights on controller pay tunneling. Journal of Financial Economics, 138(3): 777-788. Funchal, B. 2008. The effects of the 2005 Bankruptcy Reform in Brazil. Economics Letters, 101(1): 84-86. Funchal, B. and Clovis, M. 2009. Firms’ capital structure and the bankruptcy law design. Journal of Financial Economic Policy, 1(3): 264-275. Independent Evaluation Group World Bank Group    223 Funchal, B. and Monte-Mor, D. 2016. Corporate Governance and Credit Access in Brazil: The Sarbanes-Oxley Act as a Natural Experiment. Corporate Governance: An International Review, 24(5): 528-547. Ghosh, S. 2019. Loan delinquency in banking systems: How effective are credit re- porting systems. Research in International Business and Finance, 47: 220-236. Giannetti, M., Liberti, J. and Sturgess, J. 2017. Information sharing and rating manip- ulation. Review of Financial Studies, 30(9): 3269-3304. Giné, X. and Love, I. 2010. Do reorganization costs matter for efficiency? Evidence from a bankruptcy reform in Colombia. Journal of Law and Economics, 53(4): 833-864. Gnutzmann, H. and Gnutzmann-Mkrtchyan, A. 2020. The Cost of Borders: Evidence from the Eurasian Customs Union. Working Paper. Gómez, M. and Sánchez, R. 2018. Bankruptcy reforms in the midst of the Great Re- cession: The Spanish experience. International Review of Law and Economics, 55: 71-95. Gong, M., Wang, Y. and Yang, X. 2021. Do independent directors restrain controlling shareholders’ tunneling? Evidence from a natural experiment in China. Econom- ic Modelling, 94: 548-559. Grimm, M. and Paffhausen, A. 2015. Do interventions targeted at micro-entrepre- neurs and small and medium-sized firms create jobs? A systematic review of the The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix F evidence for low- and middle-income countries. Labour Economics, 32: 67-85. Hallward-Driemeier, M. and Pritchett, L. 2015. How Business is Done in the devel- oping world: Deals versus rules. Journal of Economic Perspectives, 29(3): 121- 140. Hertzberg, A., Liberti, J. and Paravisini, D. 2011. Public Information and Coordi- nation: Evidence from a Credit Registry Expansion. Journal of Finance, 66(2): 379-412. Higgins, J., Thomas, J., Chandler, J., Cumpston, M., Li, T., Page, M. and Welch, V. 2019. Cochrane Handbook for Systematic Reviews of Interventions. 2nd Edition. Chiches- ter (UK): John Wiley & Sons. Holden, P. and Pekmezovic, A. 2020. How accurate are the Doing Business indica- tors? A Pacific Island case study. Asia and the Pacific Policy Studies, 7(3): 247- 261. Hombrados, J.G. and Waddington, H. 2012. Internal validity in social experiments and quasi-experiments: An assessment tool for reviewers. Mimeo. New Delhi: 3ie. Hornok, C. 2011. Need for Speed: Is Faster Trade in the EU Trade-creating. Working Paper. Hu, J., Li, S., Taboada, A. and Zhang, F. 2020. Corporate board reforms around the world and stock price crash risk. Journal of Corporate Finance, 62: 101557. 224 Huang, H., Qu, C. and Lu, H. 2019. Can Independent Directors Effectively Monitor Controlling Shareholders? Evidence from Hong Kong. Working Paper. Huang, Y., Li, X. and Song, Z. 2021. Board reforms and audit fees: international evi- dence. Asia-Pacific Journal of Accounting and Economics, 1-20. Iwanow, T. and Kirkpatrick, C. 2009. Trade Facilitation and Manufactured Exports: Is Africa Different. World Development, 37(6): 1039-1050. Jayachandran, S. 2020. Microentrepreneurship in Developing Countries. Working Paper. Jessen, J. and Kluve, J. 2021. The effectiveness of interventions to reduce informality in low- and middle-income countries. World Development, 138: 105256. Jiraporn, P., Jumreornvong, S., Jiraporn, N. and Singh, S. 2016. How do independent directors view powerful CEOs? Evidence from a quasi-natural experiment. Fi- nance Research Letters, 16: 268-274. Jouste, M., Nalukwago, M. and Waiswa, R. 2021. Do tax administrative interventions targeted at small businesses improve tax compliance and revenue collection? Evidence from Ugandan administrative tax data. WIDER Working Paper, No. 17. Kaplan, D. Piedra, E. and Seira, E. 2011. Entry regulation and business start-ups: Evi- dence from Mexico. Journal of Public Economics, 95(11): 1501-1515. Karas, A., Pyle, W. and Schoors, K. 2015. A “de soto effect” in industry? Evidence Independent Evaluation Group World Bank Group    225 from the Russian federation. Journal of Law and Economics. 58(2): 451-480. Kirkpatrick, C. 2012. Economic governance: Improving the economic and regulatory environment for supporting private sector activity. WIDER Working Paper, No. 108. Klapper, L. 2011. Saving Viable Businesses. Public policy for the private sector note no. 328 Washington, D.C. : World Bank Group. Klapper, L., Laeven, L. and Rajan, R. 2006. Entry regulation as a barrier to entrepre- neurship. Journal of Financial Economics. 82(3): 591-629. Klapper, L. and Love, I. 2010. The Impact of Business Environment Reforms on New Firm Registration. World Bank Policy Research Working Paper, No. 5493. Klein, P. 2019. Islamic Bonds and Creditor Rights: Evidence from a Natural Experi- ment. Working Paper. Kochanova, A., Hasnain, Z. and Larson, B. 2020. Does E-Government Improve Gov- ernment Capacity? Evidence from Tax Compliance Costs, Tax Revenue, and Public Procurement Competitiveness. The World Bank Economic Review, 34(1): 101-120. Kondylis, F. and Stein, M. 2015. Reforming the Speed of Justice: Evidence from an Event Study in Senegal. Working Paper. Kraay, A. and Tawara, N. 2013. Can specific policy indicators identify reform priori- ties. Journal of Economic Growth, 18: 253-283. The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix F Lawry, S., Samii, C., Hall, R., Leopold. A., Hornby, D. and Mtero, F. 2016. The impact of land property rights interventions on investment and agricultural productivi- ty in developing countries: a systematic review. Journal of Development Effective- ness, 9(1): 61-81. Li, M. and Roberts, H. 2018. Does mandated independence improve firm perfor- mance? Evidence from New Zealand. Pacific Accounting Review, 30(1): 92-109. Li, M., Wu, J, Zhang, L. and Zou, L. 2020. Board reforms and the cost of equity: Inter- national evidence. Accounting and Finance, 60(5): 4497-4531. Li, W. 2019. Time barrier to export for OECD countries. Economics Letters, 175: 106- 112. Liberti, J., Seru, A. and Vig, V. 2015. Information, credit, and organization. Working Paper. Liberti, J., Sturgess, J. and Sutherland, A. 2020. How Voluntary Information Sharing Systems Form: Evidence from a U.S. Commercial Credit Bureau. Working Pa- per. Lichand, G. and Soares, R. 2014. Access to justice and entrepreneurship: Evidence from Brazil’s special civil tribunals. Journal of Law and Economics, 57(2): 619- 624. Lin, C., Wei, L. and Zhao, H. 2020. How Do Board Reforms Affect Debt Financing Costs around the World. Working Paper. 226 Littell, J. and White, H. 2017. The Campbell Collaboration: Providing Better Evidence for a Better World. Research on Social Work Practice, 28(1); 6-12. Liu, Y., Miletkov, M., Wei, Z. and Yang, T. 2015. Board independence and firm perfor- mance in China. Journal of Corporate Finance, 30: 223-244. Luoto, J., McIntosh, C. and Wydick, B. 2007. Credit information systems in less devel- oped countries: A test with microfinance in Guatemala. Economic Development and Cultural Change, 55(2): 313-334. Macdonald, K. 2012. The impact of business environment reforms on poverty, gender and inclusion. K4D Helpdesk Report No.898. Brighton, UK: Institute of Develop- ment Studies. Manuel, C. 2015. Is there a causal link between investment climate and growth? A review of the evidence. U.K. Department for International Development Re- search Paper. Massaro, M., Dumay, J. and Guthrie, J. 2016. On the shoulders of giants: undertaking a structured literature review in accounting. Accounting, Auditing & Accountabil- ity Journal, 29(5): 767-801. Martinez Peria, M. and Singh, S. 2014. The Impact of Credit Information Sharing Re- forms on Firm Financing. World Bank Policy Research Working Paper, No. 7013. McKenzie, D. 2009. Impact assessments in finance and private sector development: Independent Evaluation Group World Bank Group    227 What have we learned and what should we learn? World Bank Research Observer, 25(2): 209-233. Menezes, A. 2014. Debt Resolution and Business Exit: Insolvency Reform for Credit, Entrepreneurship, and Growth. Public policy for the private sector note no. 343 Washington, D.C. : World Bank Group. Michener, G. 2015. Policy Evaluation via Composite Indexes: Qualitative Lessons from International Transparency Policy Indexes. World Development, 74: 184- 196. Monteiro, J. and Assunção, J. 2012. Coming out of the shadows? Estimating the im- pact of bureaucracy simplification and tax cut on formality in Brazilian micro- enterprises. Journal of Development Economics, 99(1): 105-115. Moyer, A. and Finney, J. Rating Methodological Quality: Toward Improved Assess- ment and Investigation. Accountability in Research, 12: 299-313. Muravyev, A. 2013. Investor protection and the value of shares: Evidence from statutory rules governing variations of shareholders’ class rights in an emerging market. Journal of Law, Economics, and Organization, 29(6): 1344-1383. Oto-Peralías, D. 2017. Literature review on the effect of the ease of doing business on economic and financial outcomes. Working Paper. Petticrew, M. and Roberts, H. 2008. Systematic Reviews in the Social Sciences: A Practi- cal Guide. Kindle ed., Wiley-Blackwell, Oxford. The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix F Pinotti, P. 2008. Trust, Honesty and Regulations. Working Paper. Piza, C. 2018. Out of the Shadows? Revisiting the impact of the Brazilian SIMPLES program on firms’ formalization rates. Journal of Development Economics, 134: 125-132. Piza, C., Cravo, T., Taylor, L., Gonzalez, L. Musse, I., Furtado, I., Sierra, A. and Abdel- nour, S. 2016. The Impact of Business Support Services for Small and Medium Enterprises on Firm Performance in Low- and Middle-Income Countries: A Systematic Review. Campbell Systematic Reviews, 12(1): 1-167. Poel, K., Marneffe, W., Bielen, S., van Aarle, B. and Vereeck, L. 2014. Administrative Simplification and Economic Growth: A Cross Country Empirical Study. Journal of Business Administration Research, 3(1). Rahman, A. (2014). Investment Climate Reforms and Job Creation in Developing Countries What Do We Know and What Should We Do? World Bank Policy Re- search Working Paper, No. 7025. Rodano, G., Serrano-Velarde, N. and Tarantino, E. 2012. Bankruptcy Law and the Cost of Banking Finance. Working Paper. Rothenberg, A., Gadug, A., Burger, N., Chazali, C., Tjandraningsih, I., Radikun, R., Sutera, C. and Weilant, S. 2016. Rethinking Indonesia’s Informal Sector. World Development, 80: 96-113. 228 Sá Porto, P., Canuto, O. and Morini, C. 2015. The Impacts of Trade Facilitation Mea- sures on International Trade Flows. World Bank Policy Research Working Paper, No. 7367. Sabet, S. and Brown, A. Is impact evaluation still on the rise? The new trends in 2010–2015. Journal of Development Effectiveness, 10(3): 291-304. Sadun, R. 2015. Does planning regulation protect independent retailers? Review of Economics and Statistics, 97(5): 983-1001. Saha, A., Thorpe, J. Macdonald, K. and Megersa, K. 2021. Linking business environment reform with gender and inclusion : A study of business licensing reform in Indone- sia. K4D Emerging Issues Report No. 39. Brighton, UK: Institute of Development Studies. Schivardi, F. and Viviano, E. 2011. Entry Barriers in Retail Trade. The Economic Jour- nal, 121(551): 145-170. Schulz, M., Procher, V. and Urbig, D. 2015. The Impact of Firm Entry Deregulation on Hybrid Entrepreneurship. Academy of Management Annual Meeting Proceed- ings. Sharma, G. 2006. Competing or Collaborating Siblings? Industrial and Trade Policies in India. Working Paper. Siu, J. 2020. Formalising informal cross-border trade: Evidence from One-Stop-Bor- Independent Evaluation Group World Bank Group    229 der-Posts in Uganda. Working Paper. Snilstveit, B., Bhatia, R., Rankin, K. and Leach, B. 2017. 3ie evidence gap maps: a starting point for strategic evidence production and use. 3ie Working Paper, No. 28. Sterne, J., Hernán, M., Reeves, B., Savović, J., Berkman, N., Viswanathan, M., … Hig- gins, J. 2016. ROBINS-I: A tool for assessing risk of bias in non-randomized studies of interventions. The BMJ. 355; i4919. Ulyssea, G. 2020. Informality: Causes and consequences for development. Annual Review of Economics, 12: 525-546. Van Stel, A., Storey, D. and Thurik, A. 2007. The effect of business regulations on nascent and young business entrepreneurship. Small Business Economics, 28: 171-186. Vig, V. 2013. Access to Collateral and Corporate Debt Structure: Evidence from a Natural Experiment. Journal of Finance, 68(3): 881-928. Visaria, S. 2009. Legal Reform and Loan Repayment: The Microeconomic Impact of Debt Recovery Tribunals in India. American Economic Journal: Applied Econom- ics, 1(3): 59-81. Wang, Y., Chen, S. and Yao, J. 2019. Impacts of deregulation reform on PM2.5 con- centrations: A case study of business registration reform in China. Journal of The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix F Cleaner Production, 235: 1138-1152. White, H. and Waddington, H. Why do we care about evidence synthesis? An in- troduction to the special issue on systematic reviews. Journal of Development Effectiveness, 4(3); 351-358. White, S. and P. Fortune (2015). Business Environment Reform and Poverty: Rapid Evi- dence Assessment. U.K. London: Coffey International Development. Independent Evaluation Group. 2019. Grow With the Flow: An Independent Evalu- ation of World Bank Group Support to Facilitating Trade 2006-17. World Bank, Washington, DC. Xu, B. 2019. Permissible collateral and access to finance: Evidence from a quasi-nat- ural experiment. China Economic Review, 54: 237-255. Zeng, J., Wang, X. and Xiao, M. 2019. The impact of government property right law on collateral loans: A quasi-natural experiment based on the enactment of Chi- nese property law. International Review of Economics and Finance, 63: 273-283. 230 1  This also excludes studies which examine correlations between the DB indicators and out- come that do not relate to a specific intervention or reform. This issue relates to the attribu- tion problem discussed above; changes in the indicators can also be determined by changes in external contextual factors that are not related to regulatory reforms. 2  How well the indicators reflect the factors they measure or, at least, provide intuitive and insightful information into these issues is a methodological question that extends beyond the scope of this current review. Independent Evaluation Group World Bank Group    231 Appendix G. Summary of the Analysis of WBG Country Strategies and Frameworks IEG analyzed the extent to which the World Bank’s country strategies and frameworks were informed or motivated by the Doing Business. It also assessed whether they achieved the desired outcomes and what lessons are drown from it. To do so, IEG reviewed samples or delimited populations The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20 Appendix G of Country Partnership Frameworks (CPFs), Country Assistance Strategies (CASs) or Country Partnership Strategies (CPSs), and their corresponding Country Strategies Completion Learning Review (CLRRs), Country Private Sector Diagnosis (CPSDs), and Systematic Country Diagnostics (SCDs), pre- sented below. Country Partnership Frameworks (CPF) IEG reviewed all CPFs documents approved after FY15 (corresponding to a total of 65 countries), with a focus on those that refer explicitly to reforms measured by DB in their program focus areas and objectives or used any of the DB indicators in their results matrices. IEG’s review revealed that 45 CPFs (69%) make at least one reference to a reform measured by DB in their focus areas and objectives or used at least one of the DB indicators in their results matrices. Among these, 60% use the DB to motivate business environment related reforms, while 84% use DB indicators or sub indicators to monitor progress, and 9% to inform the design of reforms. Furthermore, 35 CPFs out of the 45 include DB indicators in their matrix framework, with 17 using the measure of distance to frontier and 24 the score for the indicator. IEG also identified whether the CPFs refer to the different DB areas, as well as to the overall EoDB. Half of the CPFs referenced the overarching EoDB category, while 36% of the CPFs mentioned trading across borders and start- 232 ing a business, respectively. Dealing with construction permits (20%), paying taxes (18%), and getting credit (16%) were more frequently referenced in CPFs than registering property (13%) and resolving insolvency (7%), while getting electricity, enforcing contracts and protecting minority investors were each referenced by only one CPF (2%). Among the CPFs that discussed DB reforms, 15 proposed a work plan that included DB-related investment interventions (4% supported by World Bank adjustment programs, 24% by World Bank investment projects, and 9% by IFC). Also, 22 of CPFs planned advisory support on DB topics (38% through WB ASA and 25% through IFC AS). Review of CAS/CPS and CLRR The review assessed how the countries are incorporating the DB report or indicators into its strategic documents, identifying the type of DB interven- tions proposed and their level of achievement. For that, IEG first rated the presence of DB in country strategies (whether a Country Assistance Strate- gy - CAS or a Country Partnership Strategy - CPS), classifying the document in a relevance range that went from “not mentioning the DB at all” to “lists reforms and includes substantial discussion related to DB”. After that, it compared the documents with the latest corresponding Country Strategies Completion Learning Review (CLRRs). Independent Evaluation Group World Bank Group    233  ating methodology for country strategy coverage of Doing Table G.1. R Business issues Rating Description None or minor Country Strategy does not mention DB reform/indicator issue at all. Country Strategy briefly mentions country’s DB ranking/indica- tors Substantial or Priority Country Strategy substantially discusses DB rankings/indicators for Reform (one or two paragraphs of contexts with multiple DB indicator or areas cited), but DB are not a top priority in the proposed reforms Country Strategy lists DB reform priorities without discussion in the document The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20 Appendix G Country Strategy lists DB related reforms + brief discussion of the DB country rankings and indicators Country Strategy lists DB related reforms + substantial discussion of the DB country rankings and indicators Source: Independent Evaluation Group Review Figure G.1. Degree of discussion of DB reform issues in CAS/CPS Source: Independent Evaluation Group Review The country strategies (CAS/CPS) sample was composed of 61 documents of lower- and middle-income countries and International Development Association or blend support-eligible countries. Lower and middle-income 234 countries together account for 73% of the reviewed documents (38 and 35% each), and more than half of the strategies reviewed correspond to Interna- tional Development Association or blend-support-eligible countries. Some 15% were FCV countries and most are in Sub-Saharan Africa (30%), East Asia and the Pacific (25%) and Latin America and the Caribbean (21%). Some 85% of the documents were elaborated between FY10 and FY2015, while the rest were from earlier fiscal years. The review revealed that 64% of the country strategies (39 countries) sub- stantially discuss or proposed DB related reform priorities. Only seven countries (Laos, Vietnam, Guatemala, Belize, Brazil, Maldives and Nepal) did not mention DB issues and 15 countries briefly discussed DB ratings and/or indicators. One country substantially discussed DB issues but did not take into consideration for reform proposals and 38 countries considered reforms measured or justified using the DB report regardless of the level of discus- sion of DB in the document. Half of the reviewed CAS/CPS used the DB as an indicator and 28% as a jus- tification. At the same time, in 28 percent of the revised country strategies, the DB report provided a justification for the proposed work program, and in 10 percent it provided a description of business environment. The CAS/CPS use the Doing Business in various ways. For example, in the 2014-2017 coun- try strategy for Poland justified the proposal that IBRD would focus on re- ducing regulatory burdens for private sector development in the areas where Independent Evaluation Group World Bank Group    235 the country is lagging the most in the Doing Business. This CPS measured results for this objective using several DB indicators such as reducing the time to start a business from 32 days to 21 days, the time to obtain construc- tion permits from 32 to 21 days, and the time to get electricity from 186 to 123 days. From the different type of DB indicators, the scores were the most often cited (52 percent of documents), rankings were cited in 33 percent of country strategies and distance to frontier in only 8 percent of the strategies. Most of the country strategies referenced the overarching Ease of Doing Business category (44%) and starting a business (43%). Meanwhile, register- ing property was mentioned in a quarter of them and trading across borders and dealing with construction permits in a fifth. In comparison, employing workers and contracting with the government were not referred to in any country strategy. Starting a business is the predominant theme in SSA and LAC, while, in ECA, it is dealing with construction permits (18%). Starting a business and paying taxes were predominant in lower-income countries. In lower-middle and upper-income countries, mentions of the different DB areas are similarly distributed. The share of resolving insolvency and getting electricity is higher in high-income countries compared to the other income group levels. Comparing FCV and non-FCV countries shows that while both focus on starting a business and trading across borders, the first group is more likely to include enforcing contracts while the second is more likely to include registering property. Roughly sixty percent of the documents proposed a work plan that dis- The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20 Appendix G cussed DB reforms and there was little evidence of cooperation between WBG institutions. Almost all of them (37/39) included advisory services and 20 included lending support. In Cabo Verde, the analysis on the interaction between institutions shows limited substantive evidence of cooperation. The CLR notes that IFC worked closely with the Bank on the investment climate and Doing Business indicators, which led to a Doing Business Task Force and adoption of a national Action Plan for investment climate reforms. The CLR also notes that information sharing between the Bank, IFC, and MIGA was adequate, but that coordination on the MSME agenda and access to financ- ing was weak (CLR, IEG 2019). The interventions identified in DB-related projects included in CAS/CPS are heavily focused on two types: improving business laws (38%) and stream- lining procedures (34%). This shows how the DB can influence at a higher through laws and regulations and also at the implementation level through the efficiency of business procedures and requirements. Unexpectedly, inter- ventions related to capacity building account for just 3 percent. In the Sey- chelles, an upstream and downstream strategy was implemented: the DPL series supported the government’s establishment of an operational online system to register companies and a virtual one-stop shop for starting a busi- ness, and the introduction of a flat fee structure for services associated with company registration. The CLR reports that the introduction of the one-stop shop and of a flat fee structure led to a decrease in the costs involved in 236 registering a business and this was reflected in a drop in DB measures of the time, cost and procedures to start a business. Effectiveness The review of the CLRR reveals that among the 38 CAS/CPS that proposed a DB related work program, 74% of them achieved or mostly achieved the DB corresponding objectives. By income level, DB-informed interventions in low-income countries were more successful with 86% of interventions achieving their objective at some degree, this number decreases to 79% for lower-income countries, 64% in middle-income countries, and 80% in high-income countries (although low in number (5)). The success rate shows a significant disparity across regions. While regions such as LAC, SSA and ECA have more than 75% of their interventions with their objective achieved, in SAR this number decreases to 67%, 50% in MNA and drastical- ly to 33% in EAP. Note that in some countries, indicators such as DB have provided support for reform. For example, the Georgian experience indicates that while the Government did many things right, the success was made possible by putting better governance front and center in its program. The various international indexes of governance, transparency, doing business, investment climate and competitiveness have played an importing support- ing role in providing a metric for government achievements and being able to communicate the value of this and the progress that has been made to the Independent Evaluation Group World Bank Group    237 electorate (CPSPR, IEG 2014). Although 74% of the 38 countries with a DB-related program achieved or mostly achieved their objectives that sought to improve the business envi- ronment, only 45% of the countries also showed improvements in DB indi- cators. The above makes us suppose that achievement of objectives does not necessarily translate into improvements in DB indicators. According to the Tunisia CPE 2005-2013, the DB survey may be an accurate reflection of business regulations de-jure, but they do not accurately reflect the very difficult de facto business environment. Tunisia’s overall ranking with respect to ease of doing business improved throughout 2014, howev- er, the evaluation raised questions about the accuracy of DB indicators in assessing the investment climate in the circumstances prevailing in Tunisia prior to the Revolution. Apparently, access of interviewers was guided by the government, directly or indirectly, towards offshore private companies (or facilitators dealing with them). Consequently, the indicators largely reflect the conditions in the offshore regime. Finally, a review of a sample of the 61 CAS/CPS shows that DB is the leading source of information to measure the business environment. In a sample of 28 countries, 60% referenced DB to measure progress on the business envi- ronment, while 40% used alternative sources such as IMF Country Reports, national records used by governments, the WB Enterprise Survey, the Lega- tum Prosperity Index, WB indicator-based reform, the WEF Global Competi- The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20 Appendix G tiveness Index, and the Fraser’s Institute Economic Freedom Index. Country Private Sector Diagnosis (CPSD) IEG reviewed all the CPSDs published in 2020 (18) and rated the DB coverage of each of them based on the methodology summarized in Table G2.  ating Methodology for CPSD Coverage of Doing Business Table G.2. R Issues Rating Description None or minor CPSD does not mention DB reform/indicator issue at all CPSD briefly mentions country’s DB ranking/indicators Substantial or Pri- CPSD substantially discusses DB rankings/indicators (one or two ority for Reform paragraphs of contexts with multiple DB indicators or areas cited), but the discussion does not translate into a reform recommendation CPSD lists DB reform priorities without discussion in the document CPSD lists DB related reforms + briefly discusses of the DB country rankings and indicators CPSD lists DB related reforms + substantially discusses the DB coun- try rankings and indicators Source: Independent Evaluation Group Review. 238 ndicators used for measuring business environment con- Figure G.2. I straints in CPSDs Source: Independent Evaluation Group Review Note: WEF=World Economic Forum; GCI=Global Competitiveness Index, WB=World Bank. Numbers do not add up to 100, as a single CPSD can identify multiple indicators or use the DB report/indicators in several ways. Most of the CPSDs (14 out of 18) substantially discussed DB related areas or proposed DB related reforms (half of the CPSDs). In 22% of the CPSDs (Indo- nesia, Rwanda, Kazakhstan, Ghana) DB issues were briefly mentioned, and in five countries (28%), DB issues were substantially discussed but this discus- sion did not translate into reform recommendations. 89% of the reviewed CPSDs used the DB indicators to provide evidence of constraints to doing business or to describe the business environment in the corresponding coun- Independent Evaluation Group World Bank Group    239 tries (72%). 61% used DB indicators as a measure of reform progress. DB in- dicators were often cited when discussing business environment constraints (89% of documents), along with information from other important WBG and global indicators (figure 1). Among DB indicators, the ranking was cited in 83% of the CPSDs and the DB category scores or other DB sub-indicator scores were present in half of them. CPSDs frequently referenced the overar- ching EODB indicator (72%). By DB area, trading across borders (78%) came up a significant constraint. It was followed by registering property (61%), paying taxes and starting a business (56% each), and dealing with construc- tion permits (44%). In contrast, employing workers and protecting minority investors were only cited in one CPSD (6% each). IEG identified constraints based on DB indicators that were heavily focused on inadequate infrastruc- ture and inefficient government bureaucracy (50% of CPSDs, respectively). Followed by access and cost of financing and inadequate regulatory frame- work (28% each of them). Inefficiencies in the tax administration, lengthy court proceedings, and informality were also frequently cited (22%). Systematic Country Diagnostics (SCD) IEG also analyzed the use and treatment of DB indicators in the Systematic Country Diagnostics (SCDs) and. A total of 63 SCDs were reviewed from a random sample stratified by income level, region, and fragility conflict and violence (FCV) status of 107 SCDs, drawn from a population of 106 countries. The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20 Appendix G The team rated each SCD based on the coverage of DB areas using a similar rating methodology than the one followed in the CPSDs review. Half of the SCDs substantially discussed and/or proposed DB related reform priorities. Only in three countries (5%) (Nepal, Uruguay, and Tajikistan) were DB issues not mentioned, and in 27 countries (43%), DB ratings and/or indica- tors were briefly discussed. In 10 countries (16%), DB issues were substantial- ly discussed but not in reference to related reforms. However, in 23 countries (36%), the SCDs considered reforms measured or justified using the DB report regardless of whether the issue was discussed briefly or substantially. Most of the SCDs referenced the overarching EoDB category (65%), while half of them mentioned trading across borders (52%), followed by starting a business, getting credit and getting electricity (40% each of them), and enforcing contracts (38%). Employing workers and contracting with the gov- ernment were referenced in eight SCDs (13%). Nearly all mentions of DB dis- cussed indicators (92%), while slightly less than half of mentions discussed reform (48%), with 46% of mentions including discussion of both. The analysis also revealed that SCDs more frequently use the DB indicators to describe countries’ business environments (27%) or to provide evidence of constraints (25%). Also, 13% of the SCDs used the DB as an indicator for measuring reform progress. SCDs tend to slightly favor citing multiple types of indicators, though DB rankings were not only the most commonly cited of all types, they were the most likely to be cited alone. 52% of SCDs cited multiple types of indicators with 35% citing an overall indicator and a DB 240 ranking, 16% adding a distance-to-frontier measure as well, and 2% citing only rank and DTF. Rank was cited most often (84% of documents) while DTF was the least cited type of indicator (19%). Over half of the 279 DB references in the sample of 63 SCDs came from just two regions, Sub-Saharan Africa (SSA) and Europe and Central Asia (ECA). When broken down regionally, SCDs from SSA were the most likely to cite DB areas (29%) with ECA a close second (24%). The rest were spread across East Asia and Pacific (19%), and Middle East and North Africa (11%), Latin America (10%), and South (6%). Independent Evaluation Group World Bank Group    241 Appendix H. Summary of Econometrical Findings Part 1: Relevance1 In this section the evaluation team provides evidence on the reliability of the DB indicators. It does so by assessing whether the DB indicators accurately measure specific aspects of the regulatory environment, whether the DB aggregate score is able to measure the quality of client countries’ regulatory The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix H environment, and whether the DB indicators accurately identify the right regulatory policy priorities. The analysis shows that DB measures for which comparable indicators from other sources could be identified generally tend to overestimate regulatory requirements. At the same time, the DB aggregate score is a reasonable measure of the overall quality of the regulatory envi- ronment, and DB indicators in general adequately identify regulatory policy priorities within their scope. How well do the DB indicators measure the regulatory environment? IEG collected several measures of regulatory environment from different sources to correlate them with the corresponding DB indicators (table H1). To ensure comparability between the DB indicators and the other data sourc- es, the team matched data by country and the year preceding the DB publica- tion date with the corresponding year of the other data source. For example, the data for DB2018 in Ethiopia was compared with the data for the Logistics Performance Index (LPI) in 2017 in Ethiopia, since the data collected for DB2018 was collected in 2017. To further enhance comparability, when the Enterprise Survey (ES) data is used, the sample includes only the responses from SMEs.2 Comparing a DB indicator with a similar indicator from a different source can help to establish the strength of their relationship. To determine the accuracy of the DB indicator in measuring the underlying regulatory phe- 242 nomenon, it is necessary to compare to measures that are reliable. The broad sample base and the orientation towards actual experience of sam- pled firms gives strong credence to the enterprise or business executive survey approaches of ES and the World Economic Forum (WEF). Likewise, LPI indicators are collected from logistics experts. Experts consulted for the corresponding DB indicators may have a more general scope of knowledge. Table H.1. Full set of indicators DB Indicator Corresponding Indicator Name Name Source Unit Collection 1 Number of Documents LPI Number Expert sur- documents to needed to vey import import 2 Number of Documents LPI Number Expert sur- documents to needed to vey export export 3 Time to get a Time to get a ES Days Firm survey permit construction permit 4 Time to import Time to import ES Days Firm survey 5 Time to export Time to export ES Days Firm survey 6 Days to get Days to obtain ES Days Firm survey electricity con- electrical con- nection nection 7 Trading across Obstacle: trade ES Likert Firm survey Borders score regulations scale Independent Evaluation Group World Bank Group    243 8 Paying Taxes Obstacle: trade ES Likert Firm survey score administration scale 9 Overall score How burden- WEF Likert Firm survey some is it for scale (face-to-face companies to and online) comply with public adminis- tration’s require- ments Source: IEG statistical analysis. Note: LPI=Logistics Performance Index; ES=Enterprise Surveys, WEF=World Economic Forum. Figure H1 shows the comparison between DB (until 2015) and LPI data for the number of documents needed to export or import. The Correlation coef- ficient for documents needed to export (figure H1.A) and documents needed to import (figure H1.B) is low, 44% and 45%, respectively. Graphically, this is evident from the low concentration of observations along the 45-degree line. Given the accuracy of the LPI indicator, the DB indicator appears to over-re- port the number of documents in both instances. A second DB indicator analyzed is the time to clear custom for imports, again from the DB and the LPI. In 2015, the DB project changed its methodology to measure this indicator3; hence, figure H2 presents the correlation between the two DB methodologies and the corresponding LPI measure. Figure H2.A shows that the methodological change in 2015 marginally improved the (low) correlation with the LPI indicator, moving it from 27% to 43%. While The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix H this change has significantly modified the distribution of the observations, it has not altered the underlying tendency of the DB indicators to over-re- port the intensity of this regulatory phenomenon. Following the approach in Hallward-Driemeier, et al. (2010)4, figure H3 presents the comparison be- tween DB indicators and their corresponding ES indicators.  orrelation between DB and LPI on number of documents Figure H.1. C needed to export or import5 a. LPI Documents to export vs DB Documents to export (DB06-15 method) Number of documents to export (DB) 244 b. LPI Documents to import vs DB Documents to import (DB06-15 method) Source: IEG statistical analysis  orrelation between DB and LPI on time to clear customs for Figure H.2. C imports using different DB methodologies a. LPI Days to import vs DB Days to import (New methodology) Independent Evaluation Group World Bank Group    245 b. LPI Days to import vs DB Days to import (Old methodology) The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix H Source: IEG statistical analysis Figure H3 and table H2 show that most of the DB indicators over-report the regulatory burden. The only exceptions are time to import and time to export when measured with the DB16-20 methodology (figure H3.B and C), which underestimate relative to the comparators. Overall, the comparison between a subset of DB indicators and similar indicators from other regu- latory environment sources consistently showed a low level of correlation, with the DB report manifesting the tendency to over-report regulatory re- quirements of time and number of documents. 246 Figure H.3. Correlations between DB and ES indicators a. ES Time to get a construction permit vs DB Time to get a permit Time (days) to get a permit (DB) b. ES Time to import vs DB Time to import (DB16-20 method) Independent Evaluation Group World Bank Group    247 Time (days) to import (DB) c. ES Time to export vs DB Time to export (DB16-20 method) The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix H Time (days) to export (DB) d. ES Time to import vs DB Time to import (DB06-15 method) Time (days) to import (DB) 248 e. ES Time to export vs DB Time to export (DB06-15 method.) Time (days) to export (DB) f. ES Days to get electricity connection vs DB Days to obtain electricity connection Independent Evaluation Group World Bank Group    249 Time (days) to get electricity connection (DB) Source: IEG statistical analysis Note: ES data refers to SMEs. Table H.2. Correlations between DB and ES indicators Corresponding Correlation DB indicator indicator coefficient Over/ Underestimate Time to get a Time to get a construction construction 23% Overestimate permit permit (ES) Time to import Time to import (DB16-20 meth- 42% Underestimate (ES) odology) Time to export Time to export (DB16-20 meth- 35% Underestimate (ES) odology) The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix H Time to import Time to import (DB06-15 meth- 25% Overestimate (ES) odology) Time to export Time to export (DB06-15 meth- 23% Overestimate (ES) odology) Days to obtain Days to get elec- electrical con- 10% Overestimate tricity connection nection (ES) Source: IEG statistical analysis Note ES data refers to SMEs and has been winsorized at the 95% percentile: Is the DB index capable of measuring the quality of the overall regularity envi- ronment? In addition to collecting data on individual aspects of the regulatory environ- ment, the DB report also produces aggregate indices and rankings that aim at measuring the overall quality of the regulatory environment. In what follows, the evaluation team assesses whether the DB aggregate score can discriminate among countries with a better or worse regulatory environment. To answer this question, the team correlated the overall DB score with an indicator of the general quality of regulatory environment from the WEF, “How burdensome is it for companies to comply with public administration’s requirements”6. As previously noted, the WEF indicator is considered more ac- curate than the (constructed) DB score, and hence it is treated as a benchmark. 250  orrelation among measures of the overall quality of the Figure H.4. C regulatory environment, DB and WEF Source: IEG statistical analysis Note: Higher values reflect a better business environment for both indicators The relationship between the DB score and the entrepreneurs’ perception of the overall quality of their country regulatory environment is shown in figure H4. The correlation has the right sign, implying that countries where the business community complains less about the quality of the regulatory envi- Independent Evaluation Group World Bank Group    251 ronment – as perceived by their own firms and reflected in the WEF indicator - receive also higher scores in the DB report, implying less regulatory burden. Figure H4 also shows that the degree of correlation among firms’ perceptions (WEF) and experts’ opinions (DB) is high - visually represented by the extent to which observations align along the reported regression line - with a cor- relation coefficient of 77%. This demonstrates that, in general, the DB score is a good approximation of the quality of the regulatory environment. Is the DB able to identify the right regulatory reform priorities? This question refers to the ability of the DB indicators to prioritize the right regulatory policy areas among those it measures. To investigate this, the evaluation team identified four comparable indicators from the DB proj- ect and the ES related to tax administration, trade regulations, access to electricity and access to finance. In particular, the ES asks questions in two different ways: one, as a Likert scale (“To what degree are each of the follow- ing an obstacle to the current operations of this establishment? None, Minor, Moderate, Major, Very Severe”); and second, as a ranking question among a list of around 10 obstacles (“By looking at the list of elements of the busi- ness environment please tell me which one, if any, currently represents the biggest obstacle faced by this establishment.”). To determine the extent to which the DB report can identify the right reg- ulatory policy priorities, the evaluation team compared the firm’s reported severity of the four regulatory obstacles from the ES with the corresponding ranking from the DB report. This analysis was conducted within each country The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix H and year for which data is available, for a total of 95 occurrences. The results for both types of ES questions are displayed in Figure H5.  lignment between DB and ES on policy priorities in four Figure H.5. A regulatory areas related to tax administration, trade regula- tions, access to electricity and access to finance Source: IEG statistical analysis Results suggest that in 20-30% of occurrences, depending on the ES ques- tion used as reference, there is a perfect alignment between the priorities identified by firms in surveys and those identified by the DB scores. In an additional 35-40%, there is a divergence of just one position in the ranking. Overall, in as little as 55% and as much as almost 70% of occurrences, the policy priorities identified by the DB are very close to those identified by entrepreneurs, indicating that the DB score is at least ‘better than average’ in identifying the more binding regulatory bottlenecks within its scope. 252 In conclusion, DB scores perform well in ranking the client countries’ quality of the regulatory environment and moderately well in identifying regulatory policy priorities within its coverage. However, their capacity to identify the relevant regulatory policy priority can be improved if the assumptions are modeled more closely to the entrepreneurs’ reported views. Part 2: Learning from Factors of Success and Failure7 This section studies how internal and external factors might influence the success of DB-related interventions. By estimating a multivariate logistic regression, the team analyses which factors are significantly associated with (or predictive of) the probability of success of DB interventions when con- trolling for other potential predictors at the project and country level. Data used in this analysis mainly comes from the database constructed by IEG for the portfolio review. First, the team identified projects related to DB. Then, from IFC AS and WB Lending evaluated projects, it identified 291 interventions and classified them as successful if they achieved or mostly achieved their objectives (World Bank ASA has no validated system of evalu- ation, so was excluded from this analysis). The team also coded standardized factors of success or failure that could have influenced the achievement of development outcomes at the project level. Those factors were classified Independent Evaluation Group World Bank Group    253 as internal or external, depending on whether they were under the Bank Group’s control or not. The internal category consists of factors related to quality at entry, project supervision and M&E considerations, while external factors include client commitment and public institutional capacity, among others. The identification of internal and external factors derives from two sources: the descriptive analysis of portfolio data on evaluated projects (appendix B) and the learning from interviews and case studies on factors of project success and failure. Interventions were also assigned a code for the DB area to which they related. Additional variables include the dollar amount devoted to DB interventions and the approval fiscal year of each project. To complement the analysis, the team also considered country level variables, such as income level and several indicators within the Worldwide Governance Indicators (WGI) -political stability, control of corruption and government effectiveness- which have proved useful in predicting project success in earlier IEG evaluations. It should be noted that the analysis is subject to two main cautions intrin- sically associated to the database construction. The first one is related to a small sample size, while the second one arises from a generally skewed distribution of successful and unsuccessful outcomes (roughly 4:1 in favor of satisfactory outcomes). In general, 82% of the interventions were classi- fied as successful, which in turn could yield insufficient variation relative to outcome. This fact is exacerbated for rarer types of reform; for instance, only 31 evaluated interventions were related to Paying Taxes, out of which 87% The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix H (27) turned out to be successful. As it is expected that the estimated stan- dard errors will be influenced by these issues, statistically significant effects presented in this section should be interpreted with caution.8 The estimated specification is shown in Equation 1, where the outcome vari- able, , is a binary variable which takes the value of 1 if the intervention of project in country and in fiscal year was successful, and 0 otherwise. Coun- try income level is represented by the vector , while stands for a set of binary variables of project’s approval fiscal year. is a vector for the percentile rating (measured on a scale from 1 to 100) of governance indicators (political stability, control of corruption and government effectiveness). The amount devoted to DB in each intervention, measured in million dollars ($US M), is represented by . is a vector of internal and external factors of success and failure, where factors considered were the ones most frequently identified in the portfolio review (84% of total factors). is a vector of DB areas9, while is a constant and is the error term. Table H3 shows the average marginal effects resulting from the estimation. Model 1 considers all the variables described in Equation 2, while Model 2 considers a restricted set of variables aimed at gaining precision in factors of success and failure that proved significant in the previous specification. The first column of Table H3 suggests that income levels, fiscal year and region are not statistically associated with the success of interventions. By contrast, 254 political stability is positively and significantly related to the outcome, im- plying that, on average, an increase of 10 percentage points in the country’s percentile rating for political stability is associated with an increase of 4.8 percentage points in the probability of success. Evidence also suggests that, on average, an additional one million dollars in intervention value is related to an increase of 0.5 percentage points in the likelihood of achieving inter- ventions objectives. Regarding factors of success and failure, the extent to which the WBG coordinates and leverages synergies across WBG institutions is positively related to the probability of success. Analytical work, proactive client engagement and follow-up and having the right mix of team expertise are the internal factors which show a positive and statistically significant relation with a successful outcome. However, while team composition is sig- nificant at the 1% level, the other two factors are significant only at the 10% level. In contrast, external factors are not significant. The estimated average marginal effect for Registering Property is negative and statistically signif- icant at the 1% level, which implies that, on average, interventions related to that DB area are likely to be less successful by 28 percentage points than interventions in other areas. In contrast, Paying Taxes is positively associat- ed to the probability of success; yet its average marginal effect (9.7 percent- age points) is significant only at the 10% level. Most of the results previously described hold when estimating the restricted specification for Model 2. The main difference is related to the MNA region’s average marginal effect, which proves significant at the 10% level and suggests that interventions in Independent Evaluation Group World Bank Group    255 the MNA region are associated with a lower probability of success (in 30.8 percentage points) than in SAR.  ultivariate Logistic Regression Output, Average Marginal Table H.3. M Effects Explanatory variables Model 1 Model 2 Income level Low 0.0524 0.0445 Lower middle 0.0548 0.0284 Upper middle and High (omitted) Year 2011-2012 (omitted) - - (continued) Explanatory variables Model 1 Model 2 2012-2013 -0.0239 -0.0359 2014-2018 0.0033 -0.0290 Region SSA 0.1506 0.1221 EAP -0.0037 -0.0083 ECA 0.1225 0.0938 LAC 0.2049 0.1615 MNA -0.3446 -0.3085* SAR (omitted) - - WDI The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix H Political stability 0.0048** 0.0050*** Control of corruption -0.0002 -0.0004 Government effectiveness -0.0027 -0.0006 Amount to DB 0.0051*** 0.0051** Factors of success and failure WBG coordination=1 0.1251** 0.1058** Complementary external interventions=1 0.0177 QAE: Analytical work=1 0.0931* 0.0879* QAE: Political and institutional analysis=1 0.0193 QAE: Risks identification=1 -0.0548 QAE: Design complexity=1 -0.0434 QAE: Suited to client capacity=1 0.0179 Superv: Flexibility of implementation=1 0.0439 Superv: Client engagement and follow-up=1 0.0848* 0.0859** Superv: Team composition=1 0.1531*** 0.1322*** Superv: Effective coordination with partners=1 0.0825 M&E: Design=1 -0.0627 M&E: Implementation=1 -0.0478 M&E: Utilization=1 0.0760 Ext: Client commitment=1 -0.0205 Ext: Agency coord. and political economy=1 -0.0499 Ext: Public institutional capacity=1 -0.0554 DB area Starting a Business=1 0.0443 Construct. Permits=1 0.0018 256 (continued) Explanatory variables Model 1 Model 2 Getting Electricity=1 -0.1238 Registering Property=1 -0.2811*** Getting Credit=1 0.0489 Paying Taxes=1 0.0966* Trading across Borders=1 -0.1032 Enforcing Contracts=1 -0.1463 Resolving Insolvency=1 -0.0031 Observations 260 270 Source: IEG econometric analysis. Note: QAE=quality at entry; Superv=project supervision; M&E=monitoring and evaluation; and Ext=ex- ternal. SSA=Sub-Saharan Africa; ECA=Europe and Central Asia; LAC= Latin America and the Caribbean; EAP=East Asia and the Pacific; MNA= Middle East and North Africa; and SAR=South Asia. * p < 0.10, ** p < 0.05, *** p < 0.01 Figure H6 plots the estimated predictive margins and 95% confidence inter- vals for the continuous variables that proved significant when estimating Model 2. This facilitates the interpretation of the estimated marginal effects at different levels of each variable and also gives a sense of the magnitude of the estimated standard errors. For instance, Figure H6.A shows how the probability of success increases with a higher country political stability rating. It also shows that standard errors are rather large at low levels of the indicator. In addition, Figure H6.B suggests non-linearities in the relation- ship between the probability of success and the resources allocated to each Independent Evaluation Group World Bank Group    257 intervention. While the first 50 $US M are associated with a relatively large marginal effect, this effect seems to decrease for the next 50 $US M and so on, reaching a precise marginal effect close to zero for higher amounts. The estimated margins for significant binary variables are shown if figure H7. Panel A suggests that when WBG coordination is absent, the probability of success is 78.6%, while it is 89.3% when this factor is present (the difference is the marginal effect of 10.6 percentage points as reported in table H3). Pan- els B-D can be interpreted in a similar way. Figure H.6. Predictive Margins (95% confidence intervals) a. Political Stability The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix H b. Amount to DB Source: IEG econometric analysis. 258 Figure H.7. Predictive Margins (95% confidence intervals) a. WBG Coordination b. Analytical Work Independent Evaluation Group World Bank Group    259 c. Client Engagement and Follow-up The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix H d. Team Composition Source: IEG econometric analysis IEG also conducted an exploratory econometric analysis to relate DB-mea- sured country reforms to development outcomes. However, it suggested that 260 it is difficult to find significant, systematic relationships between changes in DB indicators and measurable outcomes such as GDP growth, employment, FDI, trade or labor productivity. There is a high sensitivity to which variables were included (with an implicit sensitivity to omitted variables) and to small changes in model specification. For example, while one model specification showed a significant relationship between protection of minority sharehold- er rights and FDI, a slightly different model showed no significance. In some cases, DB indicators bore a counterintuitive but significant negative rela- tionship with some outcome variables (e.g., a negative relationship between resolving insolvency and employment, and between registering property and employment). Simple before and after analysis does not control for a host of explanatory factors, yet it can also be hard to specify a control group re- quired to apply more rigorous techniques. For these reasons, IEG is not offer- ing its own econometric treatment of these relationships, deferring instead to the literature. Independent Evaluation Group World Bank Group    261 Part 1 was prepared by Giuseppe Iarossi and modified by Mariana Calderon Cerbon. 1  2  IEG notes that in spite of overlapping subject matter, there are differences between the questions and respondents to DB information collection and ES information collection. These include: (i) DB collects information from intermediaries who work with firms. ES collects in- formation from senior managers or owners of firms. (ii) DB uses case scenarios to represent a specific type of firm or transaction. ES collects information on the average or most recent ex- perience of responding firms. (iii) DB attempts to reflect the experience of “domestic SMEs”. Enterprise survey samples may include large firms (with over 100 employees), but such firms’ responses were excluded for purposes of this comparison. 3  In 2015, the Trading across Borders indicator was changed to eliminate the number of doc- The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix H uments subindicator and to adapt to country comparative advantage, leading trading partner and geographic variation. 4  Hallward-Driemeier, Mary, Gita Khun-Jush, and Lant Pritchett. 2010. “Deals Versus Rules: Policy Implementation Uncertainty and Why Firms Hate it.” Working Paper 16001, NBER. Cambridge, MA USA. 5  In all graphs, outliers (the top 5% of observations) have been removed for illustration pur- poses. 6  Likert scale: 1 = extremely burdensome; 7 = not burdensome at all. 7  Part 2 was prepared by Mariana Calderon Cerbon incorporating work by Anqing Shi and Ariya Hagh. 8  To assess the robustness of statistical significance, IEG’s methods advisory team estimated a rare event corrected model, which corrects standard errors for biases introduced from a small observed sample. Although estimated standard errors were relatively large, statistical signifi- cance remained consistent. 9  Contracting with the Government, Employing Workers and Protecting Minority Investors were not included in the regression output because there were not enough observations to estimate a coefficient. 262 Appendix I. Summary of Findings related to IFC AS and WB Lending Evaluations To enrich the qualitative analysis of the factors of success and failure of WBG projects related to DB, the IEG has reviewed evaluation notes related to IFC AS projects and PPAR. Their main insights are listed in the table below. Subsequently is a summary of the deep dive on IFC AS evaluation notes.  7 Lessons from WB Lending and IFC AS Projects: Factors of Table I.1. 1 Success and Failure # Success Failure 1 Strong ownership/commitment from the DB provides only limited evidence on Government, coordinating ministry key relevance and priority of reforms needed to overcome inertia, vested inter- ests 2 Strength of interagency coordination unit Lack of focus on binding constraints key can limit development effectiveness 3 Capacity is built by learning by doing, sus- Mismatch between project complex- tainable funding mechanisms; timely and ity and client capacity undermines appropriate expertise; careful selection of success Independent Evaluation Group World Bank Group    263 contractors 4 Having the right technical expertise at the Importance of a proper framework right time and place matters. for inter-governmental cooperation across agencies and central/regional government 5 World Bank and IFC can complement each One stop shops and single windows other through collaboration. WBG organiza- need authority over the functions they tional changes at different times helped or combine, requiring process simplifi- hindered such collaboration cation and “back-office re-engineer- ing”, not just a simplified interface 6 Deeper reforms require comprehensive and Discontinuity of counterparts, regime long-term engagement. Repeat interven- change and shifts in influence of tions can be strategic or merely opportu- champions can disrupt progress nistic 7 Many IFC AS projects use other primary Global indicator standardization under indicators due to limited DB relevance, DB may be out of alignment with timeliness industry standards (continued) # Success Failure 8 Value of knowledge sharing, peer-to-peer Lender preference for immovable learning collateral, distrust and technical is- sues may constrain uptake of collat- eral registries 9 Public-private dialogue can improve owner- ship, quality of information, implementation success Source: Independent Evaluation Group. IFC AS deep dive summary The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix I The findings below were based on qualitative review of 50 evaluation notes from the IEG on IFC AS operations in the evaluation portfolio, which corre- spond to some 27% of the total number of IFC AS operations in the portfolio (184). Some The main takeaways are: » The pool of evaluated projects presents high relevance but unfortunately the evaluation approach in most cases can only test whether the intervention was relevant or not and has limited capacity to determine whether the reform proposed was a priority. » Most projects do not use DB indicators as their primary source of evidence (when used, it is as secondary evidence), but rely instead on IFC AS standard indicators. The main shortcomings are that prevent the use of DB as prima- ry indicators relate to i) lack of relevance: IFC Projects’ scope is broader or deeper than what the indicators measure ii) data collection challenges and iii) changes in DB methodology over time. » However, DB terminology has influenced the way IFC AS standard indicators are worded e.g., number of documents eliminated, days it takes, cost of pro- cures and similar. Hence, projects overlook indicators that would have been much more relevant and for which data could have easily been provided by the counterparts. » This “narrow-minded” approach to selection of indicators affects WBG’s ability to present the “business case” to Government counterparts but also diminishes WBG’s role as a “disseminator” of international best practices; 264 given that WBG puts the emphasis on collecting information that is not rele- vant per industry standards. » Impacts (understood as effects of the reforms on stakeholders) of DB reforms cannot be demonstrated due to: i) de jure versus de facto issues and ii) M&E is designed to capture private benefits but neglects public/social effects. » Improvements in DB rankings are not the focus of IFC’s portfolio. The DB draws attention and resources to the areas they measure directly but the IFC AS projects also cover themes and conduct activities that are not directly related to or would not be picked up by the DB reports. » The type of activities delivered by IFC that potentially add value to the coun- try beyond what’s measured in the DB reports include: PPDs [Public-Private Dialogues?], Capacity Building and IT improvements. » IFC’s portfolio presents instances of repeat interventions in the same coun- try over time which might suggest the intention either to deepen previously introduced reforms or to expand reforms to more areas. Hence countries with repeat interventions provide a good opportunity to explore the depth of re- forms, their evolution over time and the extent to which repeat interventions were ad hoc or strategically planned. » External factors that most affected projects’ implementation and perfor- mance are: i) weak interagency coordinating unit, ii) lack of client commit- Independent Evaluation Group World Bank Group    265 ment due to pollical economy or failure to present the “business case” iii) vested interests or shifting power and v) IT challenges iv) low capacity of counterparts. » Internal factors include technical expertise, WB-IFC coordination and WBG organizational changes. IFC AS projects previously evaluated by the IEG were found to be relevant, more clearly in terms of the WBG frameworks than in terms of national-lev- el plans and priorities. Indeed, 80% of the projects analyzed were found to be strategically relevant, but while the evidence was specific in terms of the WBG (alignment with Bank’s CPFs and regional strategies) it was not spe- cific regarding country priority. An example of such issue is in Lao (586507): “At the approval, the project documents indicate that the project supported IFC’s strategy to improve investment climate in IDA. Improvement of doing business environment was high on the government’s reform agenda. How- ever, the CRM meeting raised a relevant question that the project approval document does not explain well; why IFC supported manufacturing licensing reform as neither the Business Entry indicator of Doing Business nor the most recent Enterprise Survey showed licensing as an important binding constraint.” Furthermore, the assessment on the relevance for the private sector also counted on additional sources, such as the ICA and Enterprise Surveys. The 20% evaluated projects that were found not to be relevant have their ratings driven by external events (e.g., economic crisis) or issues relat- ed to counterparts’ lack of commitment or resistance to implementation. The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix I IEG has also repeatedly advised against using DB as the primary source of evidence or primary monitoring tool, and the review indicates DB indicators are not often used as project indicators. in several occasions. Reliance on DB data for setting baselines and targets is also problematic given changes in the DB methodology over time and the divergence between data collected at the project level and DB data collection methodologies. IEG recommended that projects define their own indicators and collect their own data. Some of the issues related to the use of DB indicators at the project level relate to lack of relevance, as IFC project is broader or deeper; challenges collect- ing data, when counterparts are responsible for generating the indicators mismatches with the DB can create problems; and to lack of consistency, as DB methodologies change over time. Even though not commonly used, the DB approach has influenced the way non-DB indicators are worded. Indeed, the IFC AS standard indicators adopt their form (e.g., number of documents eliminated), disregarding more relevant indicators for which data could have been easily provided by counterparts. When it comes to impact, the main challenges found in the project related to de jure versus de facto issues and the focus on capturing private benefits, neglecting public effects. There are instances in which reforms fail to bear fruit at the impact level, meaning, neither the private nor the public sector feel the effects of the reforms. Often this is due to laws and regulations being enacted but not implemented because of lack of complementary regulations. An example is in Comoros (580191), where the justice system did not ac- 266 knowledge the arbitration reform, leading to the disincentives for SMEs to look for that alternative. With regards to the focus solely on private benefits and a failure to capture public effects: It was noted that DB type of projects fail to articulate potential benefits for the government/operating unit (e.g. operational efficiencies, savings and similar) or the public effects of the pro- moted reforms, focusing instead on economic benefits for the private sector. This is at odds with WBG’s overall mission but also presents a missed op- portunity to show the “business case” (i.e. operational efficiency and public benefit) of the reforms for the government counterpart which in turn would increase chances of reforms’ full adoption and sustainability. As a result, at the Outcome level (adoption of changes), the fact that the WBG puts emphasis on collecting information that is not relevant per indus- try standards, affects not only WBG’s ability to present the “business case” but also its role of broadcasting international best-practices. An example is the operation on Trade logistics in the Balkans (572687), in which the reduc- tion of time and number of documents were an immediate consequence of the operation that reduced the size of samples for inspection. Instead, IEG questions the use of reduction of time and number of documents as indica- tors for this type of work. Sampling rates are indeed the most relevant indi- cators at the outcome level while the agency’s efficiency, rates of detection, (ability to detect non-compliant consignments with less screening) should be the impact. Independent Evaluation Group World Bank Group    267 The DB did draw attention and resources to its business areas measured, but the projects also covered activities that would not be reflected on DB reports. Indeed, improvements in DB rankings is not the main focus of IFC’s portfo- lio. Most (72%) projects did not have improvement of DB rankings as an ob- jective; they were categorized as DB because they used DB indicators or DB type of indicators (e.g., documents or procedures eliminated) as complemen- tary evidence, which indicates that projects were not narrowly focused on influencing DB rankings. There are projects whose primary objective was to work on areas such as investment promotion and business exit mechanisms, whose reforms would not be directly picked up by the DB. Furthermore, the same could be said of typologies of activities, work on PPDs (formal or infor- mal), capacity building activities, systems upgrades and similar, if successful, provide value added to the country beyond the mere reform and improve sustainability over time. The IFC portfolio presents instances of repeat interventions over time which suggests the deepening of previously introduced reforms or expansion to additional areas. Hence countries with repeat interventions provide a good opportunity to explore the depth of the reforms and the extent to which these repeat interventions represent a long-term commitment or strategic planning on the part of the WBG. A substantial (42) number of countries had multiple interventions. 184 IFC projects were implemented in 81 countries. Fifty two percent of the countries had two or more interventions over time, whereas 48% of countries had one intervention. A cursory review suggests The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix I that multiple projects tackle different areas and, in some instances also show evolution from a focused DB exercise to a more comprehensive intervention. In other instances, the involvement started with a multi-component inter- vention, later moving into single topic projects. IFC AS projects suffered from typical challenges such as turnover of coun- terparts, political changes, change of priorities and similar issues. Other external factors most commonly found were weak interagency coordination unit weakness. That happened when the client government’s DB reforms’ coordinating unit lacked the mandate or political power over the other par- ticipating agencies to perform a facilitator role and to enforce adoption of changes, e.g., if the unit is hosted under a Ministry that has no competencies over the reform; if the unit’s mandate was clear but they lacked capacity or resources; or when coordinated Agencies were reluctant to collaborate due to lack of trust, fear of losing competencies, political fight, old grudges, lack of communication and so on. Those issues demand a significant amount of time from IFC AS to build consensus (affecting deadlines and budget) or to understand the underlying causes for the lack of collaboration (denoting lack of proper analysis of stakeholders’ capacity during preparation). Lack of cli- ent commitment or lack of ownership was also a challenge, usually explained by the political economy that affects the approval of laws and regulations and poorly articulated “Business case” when the Government does not see the point of financing a reform that would benefit mostly the private sector. Vested interests or shift of power were linked to stakeholders’ fear of losing 268 revenue or power. An example of the last was in an automation project Leb- anon, as lawyers would be negatively affected by the project and decided to lobby against the project. Other challenges were related to IT. As many reforms require the upgrade of clients’ IT systems in order to achieve the intended impact, those are impacted by IFC’s team ability to oversee the IT upgrade, select vendors, integration with clients’ or stakeholders’ systems, secure funding and so on. A failed IT intervention can undermine project’s achievement in other areas (e.g., process simplification) in addition to presenting reputational risks for IFC if the system fails and affects government operations. An example is an IT solution developed for Food in Inspectorate in Montenegro was imple- mented only partially, in part because inspectors at the border points lacked access to internet connection. Low capacity of counterpart can also limits the ability to adopt changes, either institutional capacity (partner institution being underfunded and in lack of basic resources) and HR, or low technical capacity of staff (lack of technical knowledge, adequate training and/or high turnover of staff). These issues undermine partner institution’s ability to implement changes and its sustainability over time. An internal factor of success was technical expertise. IFC’s ability to de- ploy high quality international expertise is highly appreciated by the client governments. Unfortunately, it is sometimes difficult to find this expertise locally in low-income countries or alternatively it is difficult to find inter- Independent Evaluation Group World Bank Group    269 national experts who are willing to remain in the country for an extended period. The IFC team’s responsiveness, flexibility, and local presence along with its ability to maintain an ongoing communication and dialogue were highlighted as great strengths and a distinctive feature from other donors. Good WB-IFC Collaboration was also important. Collaboration might take the form of projects that support or complement each other either financial- ly or technically. Examples include adoption of IFC AS reforms included as a conditionality in DPLs or WB investment lending used for IT upgrades as noted earlier. The collaboration might also be informal, with the sharing of WBG experts or consultants. Overall projects reviewed show positive collab- oration, which appears in part facilitated by the Joint GP arrangement. WBG Organizational changes were also a relevant factor, as over the last 6-7 years the IFC unit in charge of enabling environment reforms has gone through several organizational changes. IFC-WB collaboration greatly benefitted from the existence of the Joint GP. There were instances in which the WBG teams worked interchangeably between IFC and WB interventions to the extent that, in projects in which IEG tried to “tease out” whether a particu- lar result was due to an IFC or WB project, the interviewed project team was unable to provide a clear answer as they were all working towards a common objective independently of who was funding what. However, organization- al changes negatively affected IFC’s project governance due to turnover of managers, competing priorities (managers or staff paying more attention to large WB operations) and even a lack of access to IFC systems by WB staff. The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix I This translated in some occasions into a lack of proactive supervision with managers failing to provide guidance and support to project teams. Project teams coped as best as they could and effects on results were minimized to the extent possible, although operational efficiency was affected. Figure I.1. PPAR chapeau report summary 270 Independent Evaluation Group World Bank Group    271 272 The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix I Independent Evaluation Group World Bank Group    273 274 The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix I Independent Evaluation Group World Bank Group    275 276 The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix I Independent Evaluation Group World Bank Group    277 278 The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix I Independent Evaluation Group World Bank Group    279 280 The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix I Independent Evaluation Group World Bank Group    281 282 The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix I Independent Evaluation Group World Bank Group    283 Source: Independent Evaluation Group. Appendix J. Review of Claims and their References Description of the Approach This note analyzes the outcomes and impacts claimed by Doing Business (DB) reports published between FY10-FY20. It also assesses whether those claims referenced rigorous sources. The identification of relevant claims fol- lowed a combination of manual codification and supervised machine learn- The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix J ing methods implemented in collaboration with IEG’s Methods Advisory Team (box J1). For each claim made, their stated channels of influence were also captured, along with their referenced sources. The present review aims to understand the claims made in terms of areas, types of reforms, and the link between the outcomes and impacts associated with the business regula- tory reforms. Additionally, it assessed rigorousness by comparing the sources used with the literature review (appendix F. Structured Literature Review). 284 Box J.1. Methodology Building on insights from the evaluation’s Theory of Change and automatic label rec- ognition techniques, the team generated a search taxonomy (key words/phrases) to identify sentences that used relevant words/phrases and were likely to claim impact. Those were: Economic growth; GDP growth; growth rate; income; investment; job; poverty; and unemployment. This initial process identified 4,085 potentially relevant sentences in the eleven Doing Business reports. To refine the initial selection and to- gether with the methods advisory team, the team created a learning sample based on manual screening of a 3% random sample (151 paragraphs). It was stratified by report and taxonomy to classify whether a sentence was relevant or not. Using the learning sample, the team ensembled three text classification models (logistic regression, support vector machine, and multi-layer perceptron) to estimate the probability of being relevant for each of the 4,085 sentences. This exercise assigned 184 sentenc- es a probability greater than an established threshold of 79% in all the three models (selected to get a manageable and useful number of results for manual review). * The team then manually screened the 184 sentences and their adjoining sentences for reference. The team eliminated those paragraphs that did not have coherent framing of reform-related effects and duplicates. That resulted in a total of 89 unique claims for which the team then manually looked for: mentions of mechanisms between reforms and their immediate outcomes, intermediate outcomes, and impact, as well as refer- enced sources. Independent Evaluation Group World Bank Group    285 * Lower levels yielded an unacceptably high level of false positives. Source: IEG Team elaboration. Areas, types of reforms, and channels of influence Starting a business was the most mentioned DB business area with 30% of the claims, followed by overall areas measured by DB (26%), employing workers (11%), and paying taxes (9%). In contrast, getting electricity was the least mentioned area, with 1% of claims (figure J1a). Regarding the types of reform described in the claims, “improve business laws or regulations” was the most frequent (44%). It was followed by “making it easier overall” (28%) and “reengineering processes” (10%). Some 7% of the claims of impact were not linked to any interventions (figure J1b).  laims’ distribution in Doing Business areas and types of re- Figure J.1. C forms a. Distribution of reform areas The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix J b. Distribution of types of reform Source: Own elaboration based on 2010-2020 DB reports Most claims did not mention clear, immediate outcomes (57%), but those which did usually referenced the cost and days of doing business and stream- lining procedures.1 » The claims including to reduce the cost of doing business (19% of all claims) as part of its channel of influence were associated with starting a business and improving business laws or regulations. In terms of intermediate out- 286 comes, were mostly associated with increasing investments, formality, and business entry, while the impacts were increasing growth and employment. The claims of reducing the cost of DB have been losing prevalence in the DB reports, as the DB2018 was the last report that mentioned them (figure J2). » The reforms associated with streamlining procedures (12%) were mainly related to starting a business, better business laws or regulations leading to increased business entry and employment increase. Intermediate outcomes were the focus of the claims (68 out of the 89 claims presented one or more intermediate outcomes). These were most frequently related to the increase in business entry, investment, and formality.2 (figure 2). » The most common channels for the claim of an increased business entry (33% of the 89) were associated with starting a business by improving laws and regulations. The immediate outcomes associated with it were streamlin- ing procedures and improving costs. The most common associated impacts were increased employment and growth. E.g., “Research provides strong evidence that reforms making it easier to start a business are associated with more firm creation, which in turn is strongly associated with job creation and economic growth” (DB2015). » The intermediate outcome of increased investment (30% of the 89) was associated with different business reform areas. Most notably, it was associat- ed with paying taxes (22% of the time). In terms of impact, it was frequently Independent Evaluation Group World Bank Group    287 related to increased growth and employment. E.g., “(…) In addition, many African economies lowered rates for the profit tax reducing its share in the total tax rate. The size of the tax cost for businesses matters for investment and growth” (DB2015). » The intermediate outcome of increased formality (20%) was mainly associ- ated with starting a business and paying taxes, with the immediate outcome of improving the cost of business entry, and with the impact of increased employment and growth. E.g., “The literature has shown that [higher] entry costs increase the size of the informal economy and decrease job creation which are likely to hurt economic performance” (DB2014).  ome of the Claims’ Most Common Channels of Influence Figure J.2. S and Selected Examples The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix J Source: Own elaboration based on 2010-2020 DB reports. Note: Dashed and thinner outline represents weaker association Some of the most common channels of influence related reforms with job creation and economic growth (figure J2). Of the 89 claims identified by IEG, 69 included at least one impact. Of these, 62% mentioned job creation and 48% mentioned economic growth. They were associated with starting a business, overall areas measured by DB, and employing workers. In terms of intermediate outcomes, they were related to increased business entry through improved laws and regulations and improved efficiency of process- es. Economic growth was also mostly associated with starting a business and overall areas measured by DB, through the increase of business entry, formality and investment, and by increasing efficiency of services. E.g., “[r] esearch provides strong evidence that reforms making it easier to start a business are associated with more firm creation, which in turn is strongly associated with job creation and economic growth” (DB2015). In most cas- es, paragraphs referred to correlations (60%) instead of causal relationships (28%). E.g., “Research provides strong evidence that reforms making it easier to start a business are associated with more firm creation, which in turn is strongly associated with job creation and economic growth” (DB2015). 288 The Rigorousness of Claims’ Sources The rigorousness assessment was guided by a comparison of the sources referenced by the claims with the ones referenced in the evaluation’s two literature reviews. The first was a desk literature review based on the Doing Business team’s bibliographic list (shared with IEG) which is organized by business area. The second, also organized by business area (excluding Labor Regulations and part of the Paying Taxes indicator), was IEG’s Structured Literature Review (SLR) (appendix F). Fourteen of the 89 claims did not mention any source, qualifying as general claims or results of the DB team’s own calculations. For example, no ref- erences are cited for the following statement: “The size of the tax cost for businesses matters for investment and growth. Where taxes are high busi- nesses are more inclined to opt out of the formal sector. Given the disin- centive effects associated with very high tax rates the continual decline in the total tax rate has been a good trend for Africa.” (DB2015). Eight of the 89 claims referenced sources within the WBG. Six of these referenced oth- er WBG flagship reports related to the DB and two referenced previous DB annual report. In DB2017 there was a reference to Women, Business and the Law 2016: Getting to Equal: “However overregulation of the labor market can discourage job creation and constrain the movement of workers from low to high productivity jobs. Stringent labor regulation has also been associ- Independent Evaluation Group World Bank Group    289 ated with labor market segmentation and reduced employment of women and youth.” DB2011 references DB2004: “For example Doing Business 2004 found that faster contract enforcement was associated with perceptions of greater judicial fairness suggesting that justice delayed is justice denied.” The other 67 claims made 116 references to peer-reviewed literature, often repeated, covering all DB areas. The business areas most frequently men- tioned by references were starting a business (41%), employing workers (15%), and overall areas measured by DB (15%). A total of 39 of the 116 ref- erences was repeated, some in different report years (e.g., Djankov, McLiesh and Ramalho 2006, was cited in the 2013, 2014 and 2020 DB reports) and some in support of different claims within the same publication (e.g., Bruhn 2011, cited four times in DB2018). Excluding repetitions, the total number of papers referenced drops to 77. When also dropping references exclusively cited in specific areas measured by the DB (8), the total of unique references made to a peer-reviewed to an indicator-specific claim drops to 69. The IEG team then performed a “matching” exercise to see whether the references mentioned were used by the IEG’s literature reviews. IEG’s Desk review (figure 3), which used a database of articles provided by DB drawn from leading journals, had four levels of evidence, ranging from limited to strong but also including a category of “mixed” where articles presented evidence both for and against a link to a particular outcome. If a finding was confirmed by multiple articles it was categorized as “strong evidence”. Only 31 of the cited papers were in the DB database. When considering specific The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix J business areas, the match dropped to 27 papers. Ten of those references were included by the IEG in its own literature desk review as providing relevant evidence on outcomes. Only 10 of the references (out of 69) were identified as relevant by IEG’s desk review, with 7 of them considered strong evidence, and validated by multiple ar- ticles. One of the 10 was confirmed by a single article, while e (were isolated) and two others were contradicted by other sources (mixed evidence). All of the seven considered part of strong evidence were related to the starting a business area, except for one to trading across borders and one on employing workers.  atching between Doing Business Claims’ References & IEG’s Figure J.3. M Desk Review Source: Own elaboration based on 2010-2020 DB reports and the Doing Business papers’ database The Structured Literature Review (SLR)3 matching exercise considered the 69 papers cited by DB reports related to specific business areas. Because the 290 SLR did not include employing workers, the references exclusively related to it (8) were also dropped from the analysis, leading to a total of 61 references. Out of these, the SLR included only 8 (13%) as meeting its criteria for rigor- ous evidence. Those were related to starting a business (4), getting credit (2) and resolving insolvency (2). Fourteen of the references that did not match SLR’s were from the business area of employing workers. The other 63 refer- ences (87%) did not meet the screening criteria of the SLR. An example is: DB2018: “Research also shows that raising the efficiency level of bank- ruptcy laws in select OECD high income economies to that of the United States would increase the total factor productivity of the former by about 30 through a rise in bank loans to large firms.” Neira (2017). The paper, related to resolving insolvency, did not meet the study design criteria. The author calibrates a model of financial intermediation and informational frictions to obtain its results. However, the SLR “excludes simple before-and-after com- parisons and simulation and forecast models.” Some references did not make it into the SLR due to other parameters of the review. It did not include articles written in any language other than English (e.g., Cardenas and Roxo (2009) which was referenced in DB 2013 but pub- lished in Spanish). It excluded articles treating cases not covered by the DB indicators, such as those dealing with special tax regimes or incentives or ex- port processing zones (EPZ’s). It also did not cover the sub-topic of tax rates (e.g., Djankov et al. (Forthcoming), which relates higher corporate tax rate to Independent Evaluation Group World Bank Group    291 lower investment/GDP ratio referenced in DB 2010). That article correlates average tax rates with other economic performance indicators, so it would have been excluded on methodological grounds, since its finding was not based on an actual reform or intervention. Indeed, many of the articles cited were excluded by the SLR on method- ological grounds because they showed general associations or correlations between two variables but did not assess the outcome of a specific reform or reform intervention (e.g. Freund and Rocha (2011)) or failed to establish a causal relationship between two associated variables (Ciccone and Papaioan- nou (2007)). 1  Claims found had a total of 51 immediate outcomes (11 of them had more than one immedi- ate outcome). 2  Claims found had a total of 96 intermediary outcomes (23 of them had more than one inter- mediary outcome). 3  The SLR was developed by the consultant Paul Fenton Villar. The Development Effectiveness of the Use of Doing Business Indicators, Fiscal Years 2010–20  Appendix J 292 The World Bank 1818 H Street NW Washington, DC 20433