2014 cy Research Wor95721 eloemp d these policy r nificant amount of pment re r k i n g Pa pe s to be learned, a sig port Public Disclosure Authoriz though much remain th e pa st ye ars. As with co nd uc ted on th ese issues over ify th e alysis has been nt also allows us to ide ck of all this research Po l i y review, tak ing sto ra- way for a new gene Research at Work 2015: Turning Insight into Impact I A in ou r kn ow led ge and helps chart the any gaps Research at Work 2015: area. n of research in this following themes: pa rti cu lar attention to the Finally, ensuring the qua lity of the supply side The rep or t pa ys nc ial sys tem in dif ferent Box O.2 continued for social services. Gra is as important as add ressing th do es th e fi na nting access to services ll Turning Insight into Impact ing acc ess . Ho w we pr ise s? nee of poor quality, or poo rly tailo • Measur d small enter ds of the extreme and chronic a chronic or transitor y conditio n) tri es dir ect ly ser ve poor households an wh ich surveys needed to measure grea poverty, this report test poo r, may lead to little or no impacts . po At coun W ho ha s ac ce ss to es increas with the e in acc frequenc ess y can minants of poverty. Improve d d is financial access? details the remainin g challeng be achieved by working directly to solve s st ho w lim ite ins ur an ce )? 28 con stra ints , than just an increase Ju , payments, h mostsuc h as sim more WORLD DEVELOPMENT REPORT 2015 countrie now splify ing enro requires ch as deposit, credit and quality of data. Althoug llment proc edures, or improving financial services (su po lic y ba rri ers to broader access? have national poverty assessme services, as opposed nts, the global todevel- imp lem enti ngof a surveys who available . Concert the edquaeffo lit obstacles and le new and prog ram. must engage with the au W hat are the chief the latest information search, as well. As novelist Jane Austen Persuasion education or is nt obs ac ce erv ss to ed MD G per for- not yet have the consis- to improve the capacity for data co as on target ifpact of access. Ho w imal actu porta opment community doesThis once wrote, “We each begin probably with a little bias  revi ewbias matic system to overcome resistance to new point • Evaluating the im th e ted by the isbuild stru ctur ed not just T in favor of view (see spotlight 2 produce h of fi rm s? Wt ha is t are sug ges tent and frequent data needed and upon to that understa every nd circumstance as follo fully country ws. The nex level t sec to on entertainment tion consultants education). exc eed s a ns poin tra int t, to th e suc groh wt as C, tha it.” Confirmation concepts, with emphas bias contributes to overconfidence is old news to political 3 examines and beh advertisers, avioraa s or finance as a co cess affects ntr h? firm growt ual rate of the nature of poverty in countrie in personal s, thewhat beliefs. evolutio People may isfail on n of to the dec recognizebetter-qu isio thatn mak ality policy makers ing poverty have also surely process of the poor. data. discovered it from th prov ed ac y’s ann Section chtoanne meels tht ugh wh rothe goaich lsim by 201 5. A cou s an d revi they ew do the experie not know they claim to know, and they own experience. nce of Chile(see Soli Public Disclosure Authorized end eh old poverty is largely dari o and v ho us over time (and whether Red Uni a nc e on poverty may fail to learn from new information chapter 10 This chapterdos , respectively.of offers a synthesis the scient h fi na e ce ss to imp lies an the imp affect development evidence on the power of the automatic tion Sec ceacyea 5 B impa ct of the latest data nt acts of thes e serv d t is the r and ial sector developme for a discussion of how these biases ices . Section 6 system nhathe n W ren a refe professionals and a survey experiment that explores con clud es by draw s wee na nc ces t asp ec ts of fi suc h inte produce t over systematic ing less behavioral ons biases. Thirty timereasonably have viewed the findings years a for the erprises? W ha y at point tG rme c - serich is more Number diat ofion surveys serv in PovcalNe A microent by 201fina5, ncial wh Figure BO.2.1 possible ices , and disc g es of confirmation bias among World Bank usses staff). people might the objectiv access to differen typ n ntr i cou es and impacts these s lan d br the P O L I CRisk Y R EAR E S oppo and C Hity REPO be R Tected to ach s will rtun e exp se r g edia de nin 13 ieve. Add tte r for ma A W O R L D B A N t,oa en su rin g su sta An ina ble exa mp le is ws. m et in E sho r K as P O eff poin ec L tiv I C Y R E S E A R C H R E P O R T e cen st t by 50 per are mo n e pov erty iqu es vices? W hat techn I all scale? a l sm Figure 1.1 Framing affects what we pay attention to and how we interpret it i on a percent, byesmo c ial ser vic 70 through So k: ending extreme poverty d oth financrate er erty re tha n ro 160 PovcalNet reference year ina has ision red credit of uce d an its pov me nt ’s le To make most decisions and judgments, we use narrow framing and draw on default assumptions and associations, which can give us a misleading A Measured Approach to Ending Poverty and Boosting Shared Prosperity prov ha t is th e go ve rn 2.Even Th e app . W targ et 140 based on limited information. arently irrational, but (psycho-) picture of the situation. seemingly irrelevant details of how a situation is presented can affect our perceptions, since we tend to jump to h Report lays po out lic iesthe broaerty to pov den acc. ess A cou ntr y is con side red off ial - conclusions • get Ad op of tin red g ucin g s? Gi ve n th at fi na nc logical behavior o he goals will take, and ial sys ive financ as point B, fallstem poor 2 The inte g inclus rlin sho rt pa of rt ofthis path. Three-year moving total ild in ked com pon sm all io an bu h a g coents in perform anc e, suc ofserrisk on ve manag ly 120 MD port Gmakes it clear that untri es eme nt with Solidar nySchool of de velopin Feeding and Primary Educati on Costs temChild per s in ma porta nt rate chal- incr eased by The assumptions und erty l e sys Number of surveys spov an im sing , the erlying classical econom h i At Loggerheads? onal policies that go of well dec rea rem ain C inst ead ce ss f ere, in g ac ics insufficiently accoun o wh 58 , ex countri pa nd es t 100 A Review a GDP e poforpu lat ion nt s to do . No mo st factors, thereby leading t for psy th comprehensive account mu men ch t Insu forBC go ranc ve usern me e s the cou ntry ’s to policies that will not, fromlenge shared 198ac ro prosperity. 9 ss to th e 200 wo rld ,6.lea Seg vin To g transfer effecresou tive, howe rces acrovess an d r, people so me 5. Box O.1 Structure 80 of com the plex report psychology, return the when adopted by peo ple with an nt ac itstio ns are gap toeq ua becllyom ing on targ et by 201 ipl es expected economic re sults. Peopleʼs behavi me te and inc guid A Measured andall illu go ve stra rn over time , from The report set ou t s to bad good pr ed by inte rpretation of their prev eriences and by current cho Surveys per year ama oductive. ious exp coun terpr to furt her sep g te those This Policy Research 60 ara Report isldstructured in three confidence aC in achieving theese goal polic t gro ies ca up, n becon we government po side r twoon licy way br state oa sdes of ni ng natu ac re ce ss, draw in in ap- wor (Bertrand, Mul lainathan and Shafir A 200 d r ia 6). And nther e are repr ntat als from of for effec absolute tive poverty those tha t are far from evidence and illustr the ating with targex amet.ple In s. the ma parts, mirroring the three 40 broad psycho aims logi cal offacttheors report. may have a stronge very attainment r impact on u C are n n reasons to sensitive in am ghgrowt to a b e the available nd on clearheaded about eac h MD G The first part 3 provides a general wea lthier peo overview ple. of the the context W the e patterns n d y of the live economic s of the poo Public Disclosure Authorized et for Driv en s average distance to be on targ by Approach to va he group’ reading for serious conceptual underpinningsvalu 20 ofes, the asp two irations goals and rence in which they live, the of extraordinary e le Rigopoo li n i hol r may shocks. or Kno wle nat dge or not.” ura l cut -off poin t to divi de the lagg ing cou ntri es into their assessment. Chapter 1 describes 0 sub -opt ima l chothe and attit ices (Ra World udes that y 2002). These are diffeJ a rent m from those of the non-poor Finally, while data and measu r Veronica Sil We and off-target and below average. Cop constructs diminish thei 08 06 04 02 00 98 96 94 92 rstand shocks, 90 throughout, r 88 possibilitie 1 86 84 extreme 82 80 inter 78 ve nal ave rage and Bank’s approach to poverty poverty by their and measurement own efforts discussed the third 20 20 et and abo 20 20 ing 20 shington, D.C. 20 19 19 19 19 19 19 19 and can be thou 19 19 19 19 Ending Poverty l cond itions, and potential ave rag e cat- ght of as synonymous of the off- targ et and abo ve assesses losseswhat of and achievement of the poverty goal pov erty (Be rtra will specifically wit ntri esredu s, thus toprtain theunce in cing half To recover from nd, Mullain athan and Shafir 2004).addresses issues But even if the poor may relate ty the bot tom make hal the f most the same thinking bias be analysis of the current es in require. of beneChapter 2 turnsBased to the from shared prosperity es monitoring the ofthe goals in greate the target,” whereas lagging countri fits Source: on data the World Bankof the non-poo PovcalNet r, the (accessed database August 2014). to context poo r face is is more for com plex between when a survey whi le maki collected, the right incentives to et.” The com put - goal, demonstrating how the goal can be evalu- there than to be a delay that of some months Chapter 5 discusses the use of average cate gory are “far from Rtheisktarg a Note: It is quite common of the non-poo r, and obli in PovcalNet. The decline ges betweenthem to mak 2010 and e2011 illustra difficult n often rtrand,available cho and Boosting n ow to achieve these goals.” d published, and when becomes it (Be Mullainathaof re con servative tha n the cut -off poin Managin ts use dO p in p Leo ortun and ated and highlighting some of reflects figure therefore also expose the the challenges fact that many d to narrowe n and surveys Shadata fir 200 collected in in 4 measuring and 2011 are 200 not 6; yet Mul global available poverty in PovcalN lainathan 2009). Th onal Development ity - interpretation. of r margins surveys erroin collected perity, r, 2011. consequently, some highlighting of the targg than a substantial decline in the number of cou ntri es as clos e to etRif iskthe fo ir r trajecto D outcomes than the non -poo and are susceptible h defi nes lagg ing eveof lo pment The second part of the report places the World r (Be rtrand, in raising Mul lainathathen and frequency Sha fir 200 4). and tim reacPro h the tect ionls, earning half a goa the required progress to Shared Prosperity Bank’s two goals in a wider context. Chapter 3 poverty estimates. Although h Public Disclosure Authorized of ured in a consistent To reduce the probability and point of 50 y, we do not use an size arb itra ry cutoff places the global poverty and shared prosperity are necessary inputs to the meas hod olog and shared ty reduction of losses and increase 5 are all less than thos 50e per cen of bene t across the MDGs, At fits goalsthe a broader insame time, discussion framework of improv of poverty and welfarein data ements that are poverty andneeded shared prosperity, he mea n gapsof nitty-gritty what ntly across countri es d population data—this off- targ et cou ntri es. analysis. to measur Ite shows poverty how andthe World shared prosper Bank’s ity choicesconsiste of ficient. Chapter 6 thus turns to pec ific Prepcutarat off poin ts to split the ion access, and ment, complementary data—populatioquality Finan trengthen capacity for affected by outliers or spuriouCop s fact ors measures should notare two options ignore the fromsan progres data of inarray possible measure ing ma y conceivably be 14 team. ss as indicators, al- that has been each achieve d in CONCEPTS, with different recentfeatures decades that provide . The DATA, purpose of power ing thisAND report parity THE to is(PPP) TW indexes th Public Disclosure Authorized loy two abs olu te leve ls of clos ene ata w , we Zealand also emp different insights. Chapter further 4 discusses improve poverty ments canpro- build differences on progres s in that thehas cost of living Agricultural identify areas where L I Care at Y R Ewit S E AhinR C H 10 and 20 Expansion, percent of bec oming onReduction, Poverty target. jections in the context of uncertainty tE S E A about particu lar eco- P O R Twhere areas growthprogres andfurther and inflation s data— REPO and RT Environment reqin the uire d Tropical to calcula Forests te the already been made and highligh PO LICY R RCH RE ssues achi dat evein on measuring in MDprac G a the World Bank adopted per tice, and form as is anc e discusse toin are nomic growth and large or unusual shocks, which support the World Bank’s pover e2013, poverty and boosting two new goalsd guide its work: Whi le kno ending Gs. extreme Unfwledortuge of poverty nat ely, risk is needed. s ofte could n has pose downside risk to achieving the goals estimates. The discussion on acco ow. h of the MD been lack ing A Measured Approach to Ending Poverty and Boosting Shared eac hin cou nd boosting ntr shared prosperity. mee t eac y to This World Bank Policy Research in deve Report lays out the 9). ndor taking on board lopi ng coun esti ma tries tes , for it isand incr easi are often ng not in adequately captured by standard ences in prices across countries onceptual underpinnings of the goals, assesses es for what 199 reaching 0, alth the oug goals h will take, and seve ral labl l, iewsand e the what in many emphasis empirical monitoringon ntri cou of the goals will require. The key area report s, such makes it clearasthatdealing0, with disease, economic models. economi Current debates around climate is particularly extensive, primar well 199 f hieving the goals will require concerted cou ntry action and dat a are transformational ava ilab policies thatlego for c n moorderre to com hold pletcountries e. If no cycl es, and natu ral haza rds. And new owle dge yond current and practices. thus report also provides a detailed and comprehensive account Theredu cing 199 0sto as sub - change and sustainability tech nolo gies receive explicit attention data have significant implication resource to policy tionanalysts, in the the late un- 198 0s are or ear grea lyhelp tly ing imp we the labl peodata Thus, e ple and info processes while face rma whe weneeded n consid to measure er global the poverty genera and l shared literatu prosperity. re on FFE, rove inkno thiswled framework. ge of pote Then- chapter demonstrates how erty estimates. they confront risk of A Measured poverty and inequality.” tial proshoc gre ksss and req uire info drespfrom port poin t unit and the n calc ula te the rate aken by the World rm Bank inonses to them. Farmers in Public Disclosure Authorized to a setof Professor iesof is the three Applied studies first component of undert Gha na for can thecountries, for upulous in exposing the fault lines in the World beBank’s inatwinccu rate goals of and if the absolute data 15poverty othe r exper- Afri nt. . MD Kno G. bywled TheseThi sge appenta studies roa ils ch mor used may a e than random just ized longitu dinal example, WFP mination 2030 and shared prosperity getting there. Honest rece and ive clearheaded spec ific about mar for-rmation through their Approach to leve l of MD G ketperinfo the p actice Grou mat ge uncertainty areion: whi sign attending le obta ificawhether the ntly poverty diff ining info eren elimination rma t from totion goal. compa Required re or reading both for serious with a contro l t are SFPs with THRs, and mob ile pho dents evenof tsdevelopment, and per theiiod they like these particular goals the not.” late st nes, progre whi ch The ss. Evidence help s them imp as the rove foundation thei r for policy design G lo b a l P r r likel ihoo doe s not ds is cap nece ture a bor ion and La eudy sam ple BIRDSALL, President,usedCenter baselin and s- e Development, for Global resp follow- up onse Washington, survey toD.C. s with changes in agri e alsodesign ANCY cultural prices and de- Ending Poverty invo ISBN lves 978-1-4648-0361-1 usin e gnow that bec info rma aus e- dat a gen man d. 24erally are yet Glo progra m not to ia c havel P r o t e c t xpo s imp allowe orta high-quality sure to d nt for report thos issu an o ers careful measurement and rigorous analysis of the current e This assessm even tsglob ent of the ability of a baliz atio itio n and for scien n, Concerns tific adva around odata Snces and measurement are often overshadowed in debates dadd ber 2014 of development and the role of policy. nt andandalposs ible eco nom ic also cris imp is. In rove und terns of the twinthe goals.rece is a fi rst crucial step to provide ofthe right the incentives spillove tor from ersta nding of many pathogens, thenyea well ras deci of to ding facilita how te the to act.community measu remen t about if the fundamental D e c em determinants and Boosting World Bank and the development Knowled to ss supportinclu ge can action to achieve these not ding be mea how sur goals.” ed they even can instelem eas two ents old dat aEconomist, memb of Chief poin asse ts, ers. pro U.K.These gre Department three projects on which for International be we dete cted and diagnosed househ TEFAN DERCON, ssm ent and judg nd Professor of Development Economics, University of Oxford - rapidlyDevelopmentto enable disease Thisconreport trol. Impargues for a different perspective—one that acknowledges the mor e, peo ple’s kno wledge of risk de- roved tech- e: nologies have also supp role d evidence plays in understanding structural change and the design Shared Prosper yals onare the info rma no. 2 (August 2012 orte ) greater colla bora tion empty if progress tion toward The they Worl canthem meeting d Bank acce Resea cannot rch ss be measured Obse rver, vol. in amo ng 27,a consistent scien d credible way. SKU The 210361 World Bank’s two new 2005 goals and 2007 of global poverty in reduction tists interna and and sharedlly poliof dis- cy policy mak and appreciates ers, as well as en- the importance of evidence in evaluating and esperity quality undert of infobetwee aken bring measurement tionn rmaissues that to is pro- the fore. By getting intoablin theg nitty-gritty the med what of ia soci district s of to info Northern rm peo ple, improving even policiesin rem over ote time. Economists rely on the availability of con- P) al equired and of camps econ inomi the data—especially the c systePader household ms. Inde and surveys ed, Lira and price and part population s of thecapacity data—this world. ort will inform both practitioners and policy makers, and help has strengthen for IDP an imp setting is somew verty analysis.” orta nt role tohat in im- edente playunprec d for studies , it doessistent and reliable data not only to motivate and assess economic CONCEPTS, DATA, theory, AND THE ailab ility , tran OHN GIBSON, Professor of spar ency , Economics,and relia bilit University of over half Waikato, New of the Zealand WFPs are in but also to monitor and evaluate economic policies in practice—and this is ized out extern al validit y since y Pro tect ion that may be relevant for risk prepara- Prot as important for poverty reduction as for other areas of economics. As the © 2015 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW Washington DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Any queries on rights and licenses, including subsidiary rights, should be addressed to the Publishing and Knowledge Division, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. Cover design: Studio Grafik Research at Work 2015: Turning Insight into Impact I i Table of Contents A Note from the Director.............................................................................1 The Leadership Role of DECRG.................................................................10 Data and Tools............................................................................................22 Evaluative Research....................................................................................28 DECRG Alumni: Where Are They Now?.....................................................34 Research Leadership Team.........................................................................36 References..................................................................................................37 Research at Work 2015: Turning Impact into Insight I 1 Section 1: A Note from the Director Asli Demirgüç-Kunt R esearch plays a crucial role in learning from past policies and thinking about the future. Without the independent scrutiny of research, conceptual and empirical foundations for policymaking would be weak, “best practices” would be emulated without sufficient evidence, and new fads and fashions would get more attention and traction than they deserve. To have impact on development policies, research must address relevant questions, answer them rigorously and persuasively, and the findings must be effectively communicated to policy makers. DECRG— the World Bank’s research department—undertakes Asli Demirgüç-Kunt and stimulates such catalytic research with the Director of Research, Research Department aim of generating new knowledge on the process of development and policies to enhance it. Although research activities occur throughout the World Bank, DECRG is the unit with a special mandate to conduct research. 2 I Research at Work 2015: Turning Insight into Impact Box 1.1: Policy Research Talks Speaker Topic Will Martin, Research Manager (former) Managing Food Price Volatility DECRG hosts monthly Policy Discussant: Anabel Gonzalez, Senior Director of Research Talks that showcase the Trade and Competitiveness Global Practice the department’s latest John Giles, Senior Economist Employment, Demographic research findings and their Discussant: Xiaoqing Yu, Director of the Social Change, and Well-Being: implications for development Protection and Labor Global Practice Avoiding Poverty among the Elderly in Aging Populations policy. These events facilitate an ongoing dialogue between Peter Lanjouw, Research Manager (former) A Measured Approach to Discussant: Ana Revenga, Senior Director of Ending Poverty and Boosting researchers and operational the Poverty Global Practice Shared Prosperity: Concepts, staff, and challenge and Data, and the Twin Goals contribute to the World Bank’s Michael Toman, Research Manager A ‘Wicked Problem’: intellectual climate. Discussant: Marianne Fay, Chief Economist for Controlling Global Climate Climate Change Change To find out more about the Aart Kraay, Senior Adviser Growth, Inequality, and Social Discussant: Marcelo Giugale, Senior Director of Welfare Policy Research Talks, including the Macroeconomics and Fiscal Management future events in the series Global Practice and videos of past events, William Maloney, Lead Economist Five Growth Mysteries visit: http://econ.worldbank. Discussant: Gerardo Corrochano, Director of in Search of a Broader org/policyresearchtalks. The Innovation, Technology, and Entrepreneurship Innovation Policy following is a list of recent Chad Bown, Lead Economist Twenty-First Century Trade Discussant: Sudhir Shetty, Chief Economist, Policy: Pushing the Limits of events in the series. East Asia and Pacific Region International Cooperation Deon Filmer, Lead Economist Schooling and Learning: Discussant: Jesko Hentschel, Senior Director, Understanding Inequalities Human Development, South Asia Region and What to Do about Them Mary Hallward-Driemer, Lead Economist Empowerment and Economic Discussant: Jeni Klugman, Director, Gender Opportunities: What Works and Development and for Whom? Philip Keefer, Lead Economist Organizing for Peace and Discussant: Joel Hellman, Global Center on Prosperity: Collective Action Conflict, Security, and Development and Development Caglar Ozden, Senior Economist Patterns of Global Migration Discussant: Shantayanan Devarajan, Chief and Opportunities for Policy Economist, Middle East and North Africa Makers Region Klaus Deininger, Lead Economist Leveraging Land Policies Discussant: Somik Lall, Lead Urban Economist, for Equity, Growth, and Sustainable Development Network Transparency David McKenzie, Lead Economist What Should We Do About Discussant: Marialisa Motta, Director, Financial Informal Firms? Why De Soto and Private Sector Development, Latin America Is Half-Right and the IRS Half- and Caribbean Region Wrong Michael Woolcock, Lead Social Development Taking Context Seriously: Specialist Expanding the Range of Discussant: Junaid Kamal Ahmad, Sector Ideas, Methods and Strategies Director of Sustainable Development, Middle to Enhance Development East and North Africa Region Effectiveness Jishnu Das, Senior Economist Delivering Education: How Discussant: Elizabeth King, Director of the Rise of Private Schooling Education, Human Development Network Changes Everything Research at Work 2015: Turning Insight into Impact I 3 By generating policy-relevant, high-quality research, DECRG seeks to shape development thinking. Being a producer of original research is also important in being able to keep abreast of advances in development thinking. Hence, for the Bank, DECRG also provides a window on all development research done outside the Bank. Given this mandate, DECRG’s audience is not only World Bank senior management and operational colleagues, but also other researchers, national policy makers, the broader development community, and the general public. DECRG seeks to reach this wide audience through academic publications, books, conferences, blogs, and other social media channels (see Box 1.1 on the Policy Research Talks, a new initiative to extend DECRG’s dissemination efforts). Despite limited resources—less than 1 percent of the administrative budget of the World Bank—Bank researchers produce a large volume of work that is of high quality and influential by academic standards, yet much more focused on development issues and developing countries when compared to the research of academic institutions (Das et al. 2013; World Bank 2012c). Much of DECRG’s interaction with operational colleagues takes place by working directly on projects with them. On average every researcher spends 30 percent of his or her time per year supporting operations, which not only facilitates the dissemination of research findings first hand but also gives researchers insight into the problems and challenges faced by client countries, hence informing and shaping DECRG’s research agenda. This continuous exchange and tacit knowledge creation, in addition to the more formal publication and dissemination channels, allows researchers to reach broad audiences both inside and outside the World Bank (see Box 1.2). 4 I Research at Work 2015: Turning Insight into Impact The discipline of policy analysis is increasingly sophisticated. In addition to research, new data and tools are required to address emerging challenges. Both research and data work in the World Bank have a long tradition of openness, with much of the research conducted jointly with outside academics and developing-country partners, findings debated publicly in a wide variety of external forums, and with Bank-generated datasets freely available and widely used and scrutinized in the broader development community (see Box 1.3 on DECRG conferences, one of the forums for such debate). This openness is integral to ensuring quality and rigor in research and data outputs, and it underlies the Bank’s broader “Open Data and Open Knowledge” policy. Moreover, as analytical and computational tools used in development policy analysis have become more sophisticated, Bank researchers have been at the forefront of developing tools to facilitate high- quality analytical work by researchers in the developing world. Box 1.2: Collaborating Department researchers spend about Youth Employment in Sub-Saharan Africa and Rise of the one-third of their time every year for Impact working with World Bank operational South: Challenges for Latin America staff. This ensures that the results of and the Caribbean as well as an research are embedded in the advice ongoing project to identify how South provided to client countries and that Asia can pursue regional energy research is relevant to the needs of security. Country-level engagements client countries. Figure 1.1 shows the cover a range of needs, from data distribution of operational support collection and analytics to background across World Bank regions. research to inform country dialogue. Examples include production of Haiti’s DECRG collaborates with operational Poverty Assessment, evaluation of a colleagues both on a region-wide South African “edutainment” project, and country level (Figure 1.2). Region- and assessment of the threat posed wide engagements often involve to Bangladesh by climate change. For Note: AFR: Africa, ECA: Europe and Central Asia, SAR: South Asia, LAC: Latin America and support to regional flagship reports more on DECRG’s collaboration with the Caribbean, EAP: East Asia and Pacific, and thought leadership initiatives. the rest of the Bank, visit: http://econ. MENA: Middle East and North Africa. Examples include reports such as worldbank.org/collaboration. Figure 1.1: Weeks of Support Provided in Figure 1.2: Share of Staff Weeks by Type of Fiscal Years 2013 - 2014 Cross-support 800 100 700 80 600 500 60 400 300 40 200 20 100 0 0 AFR ECA SAR LAC EAP MENA EAP AFR SAR MENA ECA LAC Regional Country Level Research at Work 2015: Turning Insight into Impact I 5 The work carried out in DECRG spans all sectors: with the purpose of not only maintaining core expertise in all sectors in which the World Bank is involved, but also striving to address the most important research issues within each sector. DECRG also tackles research themes that cut across sectors. One of the challenges of research is to anticipate topics sufficiently early, creating space for researchers to explore areas before demand for them is evident. Research management pays close attention to how the whole research program fits together, exploiting knowledge overlaps and synergies across sectors. Given limited resources, prioritization and selectivity are keys to ensuring that the research agenda is focused on strategic policy priorities for the World Bank and its clients (see Box 1.4 on the Research Academy, a new initiative to do more with less by incentivizing high-quality research across the entire World Bank). Box 1.3: The Convening Annual World Bank Conference on Trade, Poverty, and Shared Land and Poverty Prosperity Conference Power of Research Now in its 16th year, this annual Spurred by the World Bank’s conference brings together adoption of the Twin Goals, this The World Bank enjoys a representatives from governments, new conference brought together convening power that is based civil society, academia, the leading academics and senior policy in large part on its role as development community, and the experts to discuss how globalization private sector to discuss issues can be harnessed to reduce poverty a leader in generating new of concern to communities, land and increase shared prosperity. development knowledge. practitioners, and policy makers Participants drew on state-of-the-art DECRG both contributes worldwide. With attendance regularly research to examine the relationship above 1,000 participants, this between trade and household to and leverages off of this conference fosters dialogue and welfare. convening power by regularly the sharing of best practices on the http://econ.worldbank.org/ bringing together economists, diversity of reforms, approaches, tradepovertyprosperityconference policy practitioners, and experiences that are being implemented in the land sector Financial Sector Overview Course and representatives of around the world. Jointly organized by DECRG and the governments and international http://econ.worldbank.org/ Finance and Markets Global Practice, institutions to share new landconference2015 this week-long annual event attracts research, debate policy, officials from central banks, ministries Urbanization and Poverty Reduction of finance, and financial regulatory identify emerging issues, and Conference agencies, as well as staff involved in set the direction of future Now in its second year, the financial operations from international knowledge production. Urbanization and Poverty Reduction development organizations and conference focuses on the reality the private sector. 2014’s course, that the world is urbanizing rapidly, entitled “Challenges and Priorities for with cities today concentrating more Financial Sector Policy Makers,” was than half the world’s population. This attended by over 70 participants from annual conference, hosted by the 45 countries. The event provides World Bank, George Washington participants with a solid knowledge of University, and the Growth Dialogue, the fundamentals of key areas of the brings together academics and financial sector and engages them development practitioners to present in discussions of emerging issues of and discuss the challenges of relevance for policy makers. urbanization in developing countries. http://econ.worldbank.org/ http://econ.worldbank.org/urbaniza- financialsectoroverview tionandpovertyreductionconference 6 I Research at Work 2015: Turning Insight into Impact DECRG sets its research agenda to reflect the needs and priorities of Bank operations and the development community at large. DECRG research managers serve as focal points with Global Practices and Regions and the Director of Research is part of the Council of Regional and Network Chief Economists. Discussions take place continuously in these forums as well as at periodic meetings with regional and practice management teams to discuss research priorities. Identifying the agenda is a bottom-up exercise: topics often emerge directly from work and consultations with operational colleagues and interactions with policy makers and researchers outside the World Bank (see Box 1.5 on the Visiting Experts Program, an initiative to further strengthen this collaboration). Strategic research priorities are defined collectively by the DECRG director and research managers, under the overall guidance of the Senior Vice President and Chief Economist. The DECRG management team is jointly responsible for the design and delivery of this work program. Box 1.4: Research WINNING PAPERS 2014 WINNING PAPERS 2013 Academy All in the Family: State Capture in Harnessing Emotional Connections Tunisia to Improve Financial Decisions: The Research Academy Bob Rijkers (DECRG); Caroline Evaluating the Impact of Financial identifies and shares the Freund (Peterson Institute); Antonio Education in Mainstream Media findings of the best new Nucifora (Macroeconomics and Fiscal Bilal Zia (DECRG); Gunhild Berg Management Global Practice) (Finance and Markets Global Practice) policy-relevant research from across the World Bank Group. Highway to Success: The Impact of Report Cards: The Impact of The regular competition the Golden Quadrilateral Project for Providing School and Child Test the Location and Performance of Scores on Educational Markets is sponsored jointly by the Indian Manufacturing Jishnu Das (DECRG); Tahir Andrabi Research Director and Global Ejaz Ghani (Macroeconomics and (Pomona College); Asim Ijaz Khwaja Practices Chief Economist Fiscal Management Global Practice); (Harvard University) Jeffrey Lewis, and incentivizes Arti Grover (Macroeconomics and Fiscal Management Global Practice); Spillovers from Conditional Cash high-quality research from William R. Kerr (Harvard Business Transfers: Bolsa Familia and Crime across the World Bank. School) in Urban Brazil Laura Chioda (Latin America and Staff and consultants Progress toward the Health MDGs: Caribbean Region Chief Economist’s Are the Poor Being Left Behind? Office); João M. P. De Mello from across the institution Adam Wagstaff (DECRG); Caryn (Pontifical Catholic University of are invited to submit Bredenkamp (Health, Nutrition, and Rio de Janeiro); Rodrigo R. Soares work that makes a Population Global Practice); Leander (Pontifical Catholic University of Rio R. Buisman (Erasmus University, de Janeiro) significant contribution Rotterdam) to our understanding of Demand versus Returns? Pro-Poor development policy. Winning Targeting of Business Grants and submissions are selected Vocational Skills Training Renos Vakis (DECRG); Patrick based on technical quality, Premand (Social Protection and Labor originality, and operational Global Practice); Karen Macours (Paris relevance. For more on the School of Economics) Research Academy, visit: http://econ.worldbank.org/ researchacademy. Research at Work 2015: Turning Insight into Impact I 7 Visiting Expert Area of Research Box 1.5: Visiting Tijen Arin, Senior Environmental An expert on the use of environmentally Experts Program Economist, Environment and friendly agriculture practices to reduce Natural Resources Global Practice water pollution, Arin researched factors The Visiting Experts Program influencing farmers’ willingness to accept brings operational staff to compensation for switching practices. DECRG for up to six months, Shantayanan Devarajan, Chief Devarajan joined to examine the question and in some cases longer, to Economist, Middle East and of how to address continuing government North Africa Region failures and meet the expectations of the foster stronger links between public in the context of the Arab Spring. operations and research. For Mario Guadamillas, Practice Guadamillas examined the issue of financial more on the Visiting Experts Manager, Finance and Markets sector regulatory reform in an effort to Program, visit: http://econ. Global Practice move past the current paradigmatic worldbank.org/visitingexperts. neoclassical model, and also to help bring greater focus to the needs of developing The following is a list of recent countries in the international financial Visiting Experts. regulatory agenda. William Magrath, Lead Natural Magrath examined policy issues and law Resource Economist, Agriculture enforcement programming for natural Global Practice resources crime, where international debate has generally been driven by anecdote rather than evidence and good research. Julián Messina, Senior Messina researched firm productivity Economist, Office of the Regional and the flow of workers between sectors; Chief Economist, Latin America the determinants of the reversal in wage Region inequality in Latin America; and the impact of the migration of Latin American workers. Mahmoud Mohieldin, President’s Mohieldin explored sources of long- Special Envoy term financing to support the post-2015 development agenda, and researched field examples of leveraging the private sector and the role of financial innovation. Pierella Paci, Lead Economist, Paci’s interests lay in the intersection Poverty Global Practice of poverty reduction, growth, and jobs generation, and she sought to develop a new operational framework in this area. Joana Silva, Senior Economist, Silva extended previous research on the Social Protection and Labor long-term impact of conditional cash Global Practice transfers, and also examined the impact of international competition on the skill composition of firms. Futoshi Yamauchi, Senior Yamauchi focused on food price spike Economist, East Asia and Pacific impacts on farmgate prices and child Region nutrition status, and on the impacts of wage growth and labor shortages on small- farm agricultural production in Asia. Goran Sumkoski, Program Sumkoski focused on the links between Manager, Business Regulations infrastructure regulations and levels of Regional Lead for South Asia, IFC private investment in the corresponding Advisory Services infrastructure sectors, for example gas, electricity, roads, and communications. 8 I Research at Work 2015: Turning Insight into Impact DECRG’s current research priorities are organized into the following cross-cutting clusters: (i) Science of Delivery; (ii) Risk Management and Vulnerability; (iii) Productivity, Competitiveness, and Job Creation; (iv) Shared Prosperity and Inclusion; and (v) International Cooperation and Global Public Goods. Permeating all of DECRG’s work is an emphasis on novel data- collection, rigorous empirical work, and innovative solutions to practical policy questions. Science of Delivery research focuses on how to deliver development in a multi-sectoral fashion, including public service delivery, aid effectiveness, as well as good governance. DECRG’s work covers the public sector (e.g., the courts, customs, and land registers) and the private sector (e.g., financial services), as well as sectors with public-private partnerships (e.g., education, health, and infrastructure). It covers the labor-intensive human development sectors, as well as the capital-intensive infrastructure sectors. DECRG’s research in this area spans all aspects of the field: (a) measuring outcomes and aspects of the delivery process; (b) explaining why variations in delivery exist; (c) assessing the impacts of government reforms; and (d) designing and piloting innovative schemes to improve delivery. Risk Management and Vulnerability research recognizes that risk management can be a powerful tool for development. In recent years, the occurrence of significant economic crises and natural disasters has underscored countries’ vulnerability to systemic risks and the need to manage them. At the household or individual level, idiosyncratic risks such as losing a job, falling ill, or being the victim of a crime can have serious adverse effects and need to be managed, and they could also become systemic in certain circumstances like in the case of health epidemics. DECRG’s research at the aggregate or macro level focuses on understanding the mechanisms that shape vulnerabilities (exposure to shocks and their propagation) and on identifying appropriate risk-management policies. At the micro-level, individuals and firms confront many shocks and the department has an extensive research agenda on specific kinds of risks and on policies (such as social protection programs) and financial products to address them. DECRG distinguishes between different sources of risks and potential policy responses. Productivity, Competitiveness, and Job Creation research focuses on understanding the dynamics of economic growth and job creation and does this through four inter-related approaches: First, better measuring and understanding drivers of firm dynamics, examining their correlates with different internal and external factors within and across countries, and identifying potential areas where policy efforts could be used; Next, examining the two main avenues through which government policy actions influence private sector development and the productivity and growth prospects of a country through (a) establishing an enabling environment (general facilitation) and (b) more direct interventions (active catalyst), respectively; and finally researching the political economy of productivity policy to understand the conditions under which successful reform policies are undertaken. Research at Work 2015: Turning Insight into Impact I 9 Shared Prosperity and Inclusion research continues the World Bank’s leading role in measuring and monitoring poverty and addresses new areas of shared prosperity and environmental sustainability. The goal is to provide the international research community, civil society, and policy-makers with a better informational basis to guide poverty reduction efforts. A three- pronged approach is being pursued: (a) define and refine methods for measuring and tracking well-being and its distribution (including poverty and inequality) over time and space; (b) promote, generate, and disseminate improved data and methodological guidelines to foster the effective use of information and countries’ capacity to monitor poverty and inequality; (c) continued research on longstanding, and fundamental, questions concerning the drivers and consequences of poverty, and the impact of policies to combat poverty and inequality. International Cooperation and Global Public Goods research addresses the political economy, policy design and evaluation challenges that arise in international cooperation. Externalities across boundaries are varied and widespread. DECRG’s research analyzes a wide range of attempts at international cooperation. These include (a) regional and multilateral trade; (b) climate change and environmental security; and (c) international coordination of macroeconomic and financial policies. In addition to research, DECRG includes a team (Surveys and Methods) that focuses on data and computational tools, and another—the Development Impact Evaluation Initiative (DIME)—that focuses on impact evaluation, specifically in the context of projects implemented by the World Bank or other development organizations. These teams interact very closely with the research teams and provide important links to the rest of the World Bank and development community. Through its Surveys and Methods team, DECRG plays a valuable role in generating and maintaining original databases, as well as expanding and improving the tool kits routinely employed by policy makers and analysts in data collection and analysis. And through its impact evaluation team, the World Bank learns from its evaluative research, rigorously assessing whether development policies are effective, addressing design challenges when they are not, and disseminating knowledge on both policy successes and failures. DECRG researchers have the training and skills to inform the World Bank’s practitioners and the development community as they search for operational solutions to pressing development problems, and these two teams facilitate this interaction. 10 I Research at Work 2015: Turning Insight into Impact Section 2: The Leadership Role of DECRG ata Missing d uintile Most p repared q gross ared quin tile l s p a c e based on Least pre p of fisca indicator a , a n d a n t a te s u p port). risk cilities revenues (s ration for s a n itation fa t a g e o f ’s p re p a es action s by c debt as a percen of people s countri es , b u t e l in c lu d p u b li s th a t th e ex tent m e a c ro s in v dex show onal inco ountry le d institu- prepared R t th e c u p s a n T h is in it h n a ti m o s t for risk a omic gro esearch d produced at the ted World Bank—and w to be th e stern paration and econ k is charte be correla ple tend a and we ople’s pre ma ll s o c ia l o n fo r ris te n d s to ex te n t . P e o r th A m e ri c utions fro dex of pre parati particularly ment the work to a cer t ain produced by the in No research ecially in d contrib a te. A n in th e W o rl d Develop o n ly t ri e s (p a r ticularly e c o u n tries (esp hin r com ing the st veloped fo r cross fou department—must me coun a variety osatisfy lo w-in of demands. It exists wit ns, includ e in d e x , d e s and s e r v ic e s a h ig h -i n c least pre p a re d in al v a ri a ti o n r risk, above. Th s of asset ancial ameet ssets, e), and of substantion par pared fo the map es m e a s u re a l a n d fi nmust standards E u ro p technical e . H o w e v er,quality ly w e ll p with re p- t 2 01 4 , compris a n c a p it al, physic p re p a ration fo r ic a ), o n averag h il e is reasonab a v e r a ge risk pre p o r —hum influenc e any g major ars research A fr university. xample, It must C push na, has forward o n ly ewise, ategories ort—that vera e ye ns. For e t, Argenti capita. Lik portant c ta te su p p in c lu d e : a the frontier re g io th e e a s m e p e r o adding ries in the p o rt, and s r the index e r, and the ofhdevelopment s n eighbor to research, a r le veboth l of incby unt cia l s u p n t in dica to r s fo e d 1 5 an d o v o f w il e it h a v in ga s im il th a n other co . This k. The co mp o n e e p o p u lation a g ; th e ptoro p or available the tio n a r a tidata o n des on development p it e ri s k p re p aratitopics and o n c o m e p er capita ooling fo r th (human c apital) ess as better higher in income total sch m e a s le s n d a n in dex of acc advancing research E th io pia h techniques. o r re And la tively it must o vachieve e r a n d above r , a r , k. n iz ation rate fo ,0 0 0 in net asset s e n t o f the work - g io n w ith simila a n c e o f policies p a r a ti on for ris m u with less than $ 1 ); the pe rc influence n ofby expanding re the e imp base o r t of policy-relevantmining p re useholds fi n a n c ial assets d th e proportio u n d e rscores th s o u rc e s , in deter d heme, a n re hysical an ension sc knowledge trusted” (s ocial for the World e l an d a c cess to Bank’s operational staff and finance (p to a p c a n b e d le v s of the o c o n tribute n e ra l, p eople for s to impolicy p rove makers in developing countries. rd in a l propertie ce w h that “in g e with acce s aintains th e c a tion of s t atin g u latio n th u s m in th e construc s p , d pondent po easure icato rs gy us e e p e rc e nt of the o f th e eight ind th e m ethodolo in c re a se in its m d th e averag e in pa rt so that a n pport); an mber: IBR D 40097. What which is th x, evidence is there pproach follows that World an sformedresearch Bank p nu inde his a r is tr e W D R 2014. Ma ro an d one. The co m p onents. T a ry, e a ch indicato 013 for th n z e ss th e es s rce: Foa 2 le d to rang e betwee e of rankin is gs a cro meeting astruzzi 20 these 10). If necgoals? A range of metrics—from r is re s c a a n ave ra g y, an d M a to in g K raa ach ind ic an simply b e Kaufman n , web traffic to citation counts and survey data— d ic ato rs , rather th ce In d ic ators (see n erna n wide Gov demonstrate that DECRG research is in demand, gis he World im p ro vement. 0 9 cr isis—includin epresents a n influential, and is relevant. 08 – d he mids t of the 20 , fiscal st imulus, an f world in t rge financial firms helped calm e ss ed sale o f la bailouts o iods of monetary e a sing — ve and d is tr - ses may ha in surance, e ll as long extended p er ese re sp on y t, lac k o f -term a s w o rt-run, th luding substantiall su b st a n tial short ef fe c tiv e ly become t he sh markets in er-term effects, inc incentives for fi- —has n seholds ca more vulnerable ng rse fects: p o o r h o u m negative lo lic debt and perve m aking the ndertake pub . d in p o v e r t y, ss ab le to u increased ’ r isk taking o cks and le 30 Similarly, while stit u tio ns re neg at iv e sh . nancial in fo r im provement nts around the entures me s p o n s es by govern ping re Research at Work 2015: Turning Insight into Impact I 11 The Demand for Bank Research DECRG produces a wide range of research outputs, including working papers, books, datasets, policy reports, and peer-reviewed journal articles. Between 2011 and 2013, DECRG on average produced 155 peer-reviewed journal publications per year. The production of research begins with the Policy Research Working Paper (PRWP) series, established in 1988 as the primary vehicle for the dissemination of new research produced by the World Bank. The series is a stepping-stone to publication in a peer-reviewed journal, which ultimately certifies the quality of the research. This body of work then frequently feeds into outputs targeted to a broad audience of policy makers and other development stakeholders, whether in the form of a World Development Report, Policy Research Report, or other synthesis document.i Given the central role of PRWPs in the production of research, download statistics for this series provide one useful indicator of overall demand for research. PRWPs are disseminated through multiple channels, including two of the world’s largest online aggregators of social science research— Research Papers in Economics (RePEc) and the Social Science Research Network (SSRN)—and through the World Bank’s own websites. Data from both RePEc and SSRN indicate high demand relative to comparator working paper series, while data from the World Bank’s own websites indicate particularly high demand for work authored by DECRG. RePEc provides an online bibliographic database of research in economics and related disciplines. As of October 2014 RePEc listed 1.4 million pieces of research by over 35,000 authors.ii Download data from RePEc indicates that PRWPs are one of the most frequently downloaded series on a per paper basis compared to other major working paper series included on the site. PRWPs have been downloaded an average of 229 times per paper— placing the Bank’s working papers ahead of other major series such as the National Bureau of Economic Research, the Center for Economic and Policy Research, and the IMF Working Paper series, among others (Figure 2.1). Figure 2.1: Downloads per World Bank Paper via RePEc National Bureau of Economic Research Source: RePEc, accessed September 29, 2014, Center for Economic and Policy Research http://logec.repec.org/scripts/seriesstat.pf. IMF Note: Based on lifetime downloads of all items Institute for the Study of Labor (IZA) in each working paper series. Includes only CESifo Munich series with 1,000 or more working papers. RePEc directs site visitors to copies of Policy Munich Personal RePEc Archive Research Working Papers hosted on the World 0 50 100 150 200 250 Bank’s website. The numbers reported in this figure reflect only downloads generated via RePEc-generated requests. 12 I Research at Work 2015: Turning Insight into Impact A similar story emerges upon examination of data from SSRN. As suggested by its name, SSRN focuses on dissemination of research work in the social sciences, and contains a collection of over 473,000 downloadable documents. The PRWPs once again emerge as the most frequently downloaded working paper series on a per paper basis against the major comparators available in SSRN (Figure 2.2). PRWPs have been downloaded an average of 223 times per paper, compared to 212 times per paper for the New York Federal Reserve, the next closest series. The World Bank also far outstrips other multilateral development banks (i.e., the Inter-American Development Bank and the Asian Development Bank), which together average 57 downloads per paper. Downloads from the World Bank’s own websites extend the story, and also allow us to differentiate downloads of PRWPs produced by authors from different parts of the World Bank. Over the period 2008–2013, over 2,000 PRWPs were produced, with about one-third of these authored or co-authored by DECRG staff. Across the full set of PRWPs, papers were downloaded directly from the World Bank’s own websites an average of 379 times—but with a large differential depending on authorship. DECRG- authored or co-authored working papers were downloaded 72 percent more often on average than non-DECRG authored papers. While DECRG-authored working papers were more frequently downloaded on average, it is possible that this could be entirely due to a few “big hits”—a handful of DECRG-authored papers that generated great numbers of downloads. A closer analysis of working paper downloads by percentile demonstrates that this is not the case. Not only are DECRG-authored working papers more frequently downloaded on average, they are also more downloaded across the spectrum. DECRG-authored working papers are downloaded more frequently than non-DECRG working papers at every percentile of the distribution of downloads (Figure 2.3). At the 1st percentile, the least downloaded DECRG-authored working papers were downloaded 55 percent more often than non-DECRG-authored working papers (79 versus 55 downloads). And at the 99th percentile, DECRG-authored working Figure 2.2: Downloads per World Bank Paper via SSRN New York Federal Reserve Source: SSRN, accessed September 29, 2014, Federal Reserve Board http://www.ssrn.com/en/index.cfm/rps/. CESifo Note: Based on lifetime downloads of all items IMF in each working paper series. Includes only series with 1,000 or more working papers. The European Central Bank category “Other Multilateral Development Institute for the Study of Labor (IZA) Banks” is composed of the Inter-American Development Bank and the Asian Development Other Multilateral Development Banks Bank. National Bureau of Economic Research Centre for Economic Policy Research 0 50 100 150 200 250 Research at Work 2015: Turning Insight into Impact I 13 3500 Figure 2.3: Document 3000 Downloads by Percentile, 2008-2013 2500 2000 Note: Original analysis of downloads of World Bank ESW due to Doemeland and Trevino 1500 (2014).The figure reflects updated analysis 1000 for the 2008-2013 timeframe. Downloads of working papers based on authors’ own 500 calculations. 0 p1 p5 p10 p25 p50 p75 p90 p99 Working Papers (No DECRG Author) Working Papers (DECRG Author) Economic and Sector Work (ESW) papers were downloaded more than twice as often as non-DECRG-authored working papers (3,215 versus 1,540 downloads). Regardless of authorship, the PRWP series outperformed World Bank Economic and Sector Work (ESW) in terms of downloads, both on average and at every percentile. ESW primarily consists of operational reports written by World Bank country economists in support of the policy dialogue with their client countries. Between 2008 and 2013, ESW reports were downloaded 148 times on average, with the median ESW report downloaded just 48 times (Figure 2.3). The corresponding figures for PRWPs were 379 and 260. The Influence of Bank Research While download data reflects the general demand for World Bank research, it does not provide a direct indication of the influence that this work has on the broader research discourse. For this, the tools of bibliometric analysis are required. Bibliometrics provide a quantitative assessment of the influence of published research through analysis of citations. The availability of large databases such as SCOPUS, an online bibliographic database containing abstracts and citations from some 21,000 journals, has vastly expanded the ability to carry out this kind of investigation. Bibliometric analysis provides a range of tools to assess influence—reflecting different judgments about how best to value different types of citations. For example, it is clear that a citation in a top-ranking journal should carry more weight than a citation in a lesser-known journal focusing on a narrow subfield. But there is no agreement on exactly how to rank journals or weigh citations, which has given rise to a multitude of approaches to measuring influence (Engemann and Wall 2009). Given the lack of agreement, two different approaches to analyzing the influence of DECRG research are considered. The first uses a neutral metric: an unweighted average of the number of citations per paper, benchmarked against comparator institutions. The second draws on the well-known rankings produced by RePEc, which synthesizes a broad set of underlying bibliometric indicators that adjust for many factors such as quality of the journal in which a citation appears. 14 I Research at Work 2015: Turning Insight into Impact Figure 2.4: Average Number Harvard University of Citations per Paper World Bank (DECRG) (Articles Published University of Chicago between 1995 and 2014) Massachussetts Institute of Technology Note: Citation data drawn from the SCOPUS Princeton University database, which covers 21,000 titles from all New York University journals. http://www.elsevier.com/online-tools/ scopus Brown University Yale University Columbia University International Monetary Fund (IMF) Cornell University World Bank (Other) European Bank for Reconstruction and Development (EBRD) London School of Economics University College of London Stanford University United Nations Development Program (UNDP) Oxford University Inter-American Development Bank University of California, Berkeley African Development Bank Asian Development Bank 0 5 10 15 20 25 30 35 40 How does DECRG-authored research stack up in terms of citations of its work? DECRG is clearly influential when compared against a broad set of academic and development institutions. Peer-reviewed journal articles and handbook chapters published by DECRG between 1995 and 2014 have been cited an average of 33 times in peer-reviewed journals, placing DECRG second only to Harvard University and far ahead of many other well-known research institutions (Figure 2.4). By this metric DECRG research clearly has wide influence. DECRG research necessarily tackles both big questions of wide interest and less attention-grabbing questions that are still of vital importance to the World Bank’s clients. Nevertheless, an analysis of citations by percentile demonstrates that the full spectrum of DECRG-authored research is cited frequently relative to comparators (Figure 2.5). Consider an article at the 50th percentile of the distribution of citations for a given institution. This “typical” article for DECRG will have 12 citations, compared to 10 citations for Harvard University and 4 citations for the IMF. This puts DECRG in 1st place when ranked against comparator institutions. DECRG does well at the high end as well. An article at the 90th percentile for DECRG is cited 81 times, putting it in third place behind Harvard University (93 citations), and Princeton (83 citations). Research at Work 2015: Turning Insight into Impact I 15 (Articles published between 1995-2014) Figure 2.5: Number Institution 25th 50th 75th 90th 99th of Citations across a percentile percentile percentile percentile percentile Range of Percentiles Harvard University 2 10 36 93 433 Source: SCOPUS database, University of Chicago 2 9 31 78 352 accessed October 3, 2014, World Bank (DECRG) 3 12 33 81 291 http://www.scopus.com. Note: Institutions are color- Massachusetts Institute of 2 10 33 78 278 coded in each column based on Technology their rank within that percentile. Princeton University 1 9 33 83 297 The highest-ranking institution New York University 1 8 28 70 293 is coded deep green, the median institution is coded mid- Brown University 2 9 28 64 249 yellow, and the lowest-ranking Yale University 2 7 22 54 275 institution is coded deep red. Columbia University 1 6 21 55 225 Others institutions are coded at intermediate colors in proportion Cornell University 1 5 18 48 185 to their distance from the International Monetary Fund 1 4 14 38 215 median. World Bank (Other) 1 4 15 39 285 EBRD 0 5 23 46 154 London School of Economics 1 4 16 41 203 Oxford University 0 4 13 35 155 Stanford University 1 4 15 36 146 University College of London 1 4 14 37 132 UNDP 0 2 7 43 171 Inter-American Development Bank 1 3 12 29 110 University of California, Berkeley 0 2 8 19 97 African Development Bank 0 1 5 17 116 Asian Development Bank 0 0 3 9 65 The more sophisticated set of rankings produced by RePEc provides a second way of measuring the influence of World Bank research. Instead of simply counting unweighted numbers of citations, RePEc aggregates across 32 different metrics. These metrics cover everything from number of works produced to citation counts, journal page counts, and RePEc web traffic. Over 40,000 registered authors are ranked on these 32 metrics, and these rankings are in turn used to rank the institutions to which the authors are affiliated. Out of more than 6,700 institutions, DECRG ranked number 12 in economics research overall, and ranked first in development economics (Box 2.1).iii This placed DECRG ahead of Brown University, Harvard University, and the London School of Economics in the domain of development economics. DECRG’s high ranking as an institution is of course driven by the high ranking of many of the researchers who belong to it—and, in fact, a total of 19 DECRG researchers are ranked within the top 10 percent of the field of development economics. Since DECRG researchers work across many fields that are important for economic development, their expertise also reflects this diversity: 28 of them are ranked in the top 10 percent of their respective fields. DECRG researchers also rank in the top 1 percent of five different fields: Agricultural Economics, Banking, Human Migration, Health Economics, and Positive Political Economics. 16 I Research at Work 2015: Turning Insight into Impact Box 2.1: RePEc/IDEAS: RANK INSTITUTION Top 10 Institutions in the Field of 1. World Bank Group/DECRG Development Economics 2. Brown University (Economics Department) (as of October 2014) 3. Harvard University (Economics Department) 4. London School of Economics Source: RePEc Top 10% Institutions and Economists in the Field of Development, 5. Yale University (Economics Department) accessed November 11, 2014, https://ideas. 6. National Bureau of Economic Research repec.org/top/top.dev.html. 7. International Food Policy Research Institute Note: The World Bank and DECRG are both scored in REPEC’s ranking methodology. 8. Massachusetts Institute of Technology (Economics Department) However, REPEC does not separately rank 9. International Monetary Fund institutions and their sub-entities, as in the case of the World Bank vis-à-vis DECRG, but 10. Oxford University (Economics Department) DECRG’s score places it just behind the World Bank as a whole (Zimmermann 2012). The Relevance of Bank Research While World Bank research is in demand and influential, this is not enough. It must also be relevant to the issues faced by the developing countries that the World Bank aims to serve. Given the broad range of challenges that developing countries experience, there is no single metric that allows us to gauge relevance. However, a number of metrics, each reflecting some aspect of relevance, suggest that DECRG research fills a gap in addressing the research needs of developing countries. Map 2.1: Total Publications per Million Citizens Source: Das, Do, Shaines, and Sowmya, 2013. Research at Work 2015: Turning Insight into Impact I 17 One way of judging the relevance of World Bank research to the needs of developing countries is to consider how the geographical focus of Bank research stacks up against other producers of research. It turns out that startlingly little empirical research is done on low-income countries. For example, over the 20 year period 1985–2004, just four papers were published in the top 202 economics journals on Burundi, nine on Cambodia, and twenty-seven on Mali (Das et al. 2013). To put that into perspective, some 36,649 empirical economics papers on the U.S. were published over the same timeframe in the same journals (Map 2.1). What is driving this distribution of research publications? Das et al. (2013) find that the most important factor is the per capita GDP of a country, which alone accounts for 75 percent of the variation in per capita publications across countries. But for the World Bank this “research-wealth relationship” does not hold. Instead, the distribution of World Bank research is clearly tilted in favor of Low and Lower-middle Income Countries (Figure 2.6). More than one-third of the World Bank’s journal articles have been on Low-income Countries, compared to only 11 percent on High-income Countries. Among non-Bank authors the picture is reversed: around 8 percent have been on Low-income Countries and over 70 percent on High-income Countries. The World Bank alone accounts for 7 percent of the total global output of research on Low-income Countries. These findings underline the point that DECRG fills a gap that is left by purely academic research. It is also possible to gauge the relevance of DECRG research via its impact on other World Bank knowledge products—specifically, Economic and Sector Work (ESW) reports targeted towards policy makers. A study by Doemeland and Trevino (2014) analyzed five years’ worth of download statistics for ESW. They find a great deal of variation in report downloads, with nearly one-third of ESW reports never having been downloaded a single time. 80% 8% Figure 2.6: Geographical 70% 7% Distribution of Research by 60% 6% Country Income Classification 50% 5% Source: Authors’ own calculations based on 40% 4% Das, Do, Shaines, and Sowmya, 2013. 30% 3% Note: World Bank country income classification 20% 2% is as of 2000. 10% 1% 0 0% Low-income Lower-middle Upper-middle High-income income income Country Grouping's Share in World Bank's Work Country Grouping's Share in Other Institutions' Work World Bank's Share in Total (Right-hand Side) 18 I Research at Work 2015: Turning Insight into Impact Figure 2.7: Downloads of Policy 1,000,000 Reports by Percentile, 2008- 100,000 2013 Note: Doemeland and Trevino (2014) do 1,000 not report results at the departmental (i.e., DECRG) level. The figure reflects data for the 100 Development Economics Vice Presidency (DEC), the administrative unit to which DECRG 10 belongs. 1 p1 p5 p10 p25 p50 p75 p90 p99 Max Reports (No DEC Support) Reports (DEC Support) Doemeland and Trevino (2014) also looked at the relationship between downloads and cross support, i.e., staff time provided across organizational units to support a project, and their findings were stark: DEC was the only vice presidential unit whose cross support was linked with greater downloads and greater citations. DEC-supported reports received 302 downloads on average—more than double the 129 downloads per report for non-supported reports (Figure 2.7). This relationship of increased downloads for DEC-supported reports also held at every percentile. This record of downloads points to the policy-driven nature of DEC research, which has helped enhance the relevance of ESW reports specifically geared towards a policy making audience. This policy relevance is also one of the policy-driven nature of DECRG research is frequently cited in major media outlets (Box 2.2). Box 2.2: Research in the Media The work of DECRG is frequently the go-to source for major media outlets around the world covering development economics and related areas of work, with numerous references in major outlets like The Economist, Financial Times, or Bloomberg. Journalists often turn to cross-country data produced by DECRG on topics where no other authoritative data is available or to innovative policy research that overturns received wisdom. The following headlines represent a small World Bank: Emerging nations sample of the recent press coverage have borne brunt of protectionism received by DECRG’s work. Research at Work 2015: Turning Insight into Impact I 19 Box 2.3: Policy managed by a small team in DECRG Reports: Translating for the last 20 years and serves as the Research Insights into basis for all global poverty estimates Finance for Development Impact and projections. At the heart of the Policy Research Report is the extensive set of research papers written by Martin All? Ravallion and Shaohua Chen that When new policy findings are provide base estimates, updates, and published in a peer-reviewed journal, explanations of global poverty counts. Policies and Pitfalls the work of DECRG is still only The report also draws from DECRG in Expanding Access partly done. Researchers devote research products that provide case considerable time and effort to studies of poverty measurement for synthesizing their findings into country policy. publications that are accessible to Policy Research Report: Finance policy makers and the public at The report points the way forward on for All? Policies and Pitfalls in large. Some of the most significant the World Bank’s goals in a number Expanding Access (2007) of these are the Policy Research of respects. Reaching the goal of Report series and the Global Financial reducing the percentage of the DECRG has done significant research Development Report series, both world’s population living beneath on access to finance and financial of which have turned large bodies the poverty threshold to 3 percent is inclusion for the last decade, of research into policy-oriented achievable, but highly aspirational. examining issues like the importance publications that have spurred action Consequently, a greater focus on of access to finance for development, both within the World Bank and in the shared prosperity is required, as identifying and filling data gaps, World Bank’s client countries. the two goals mutually reinforce and providing practical, evidence- each other. And the report provides based advice on policies that guidance on how and where support healthy financial inclusion. investments in data systems should This research was synthesized in the POLICY RESEARCH REPORT be made, including in the kind of Global Financial Development Report ding extreme poverty high-quality household survey data 2014: Financial Inclusion. The report A Measured Approach to Ending Poverty and Boosting Shared Prosperity port lays out the oals will take, and makes it clear that policies that go well rehensive account d prosperity. produced by the Living Standards built on a policy agenda first laid out A Measured Measurement Study (LSMS). in the 2007 Policy Research Report absolute poverty earheaded about ing for serious t.” Approach to ton, D.C. sis of the current right incentives to Ending Poverty Finance for All?, which highlighted and Boosting both the relevance of the topic and achieve these goals.” Development Shared Prosperity the large data gaps in this area. in a consistent duction and shared ty-gritty of what pulation data—this then capacity for CONCEPTS, DATA, AND THE TWIN GOALS aland The Global Financial Inclusion (Global s in measuring and verty and boosting in taking on board Global Financial Development RepoRt 2014 d the emphasis on Findex) database—produced with der to hold countries urce to policy analysts, Global Financial Development Report 2014 is the second in a new World Bank series. It contributes to financial sector policy rty and inequality.” debates, building on new data, surveys, research, and country experience, with emphasis on emerging markets and developing Development RepoRt ssor of Applied economies. The report’s findings and policy recommendations are relevant for policy makers; staff of central banks, ministries of significant funding from the Bill and Global Financial finance, and financial regulation agencies; nongovernmental organizations and donors; academics and other researchers and analysts; and members of the finance and development community. This year’s report focuses on financial inclusion—the share of individuals and firms that use financial services—and ISBN 978-1-4648-0361-1 demonstrates its importance for reducing poverty and boosting shared prosperity. It also underscores that the objective is not financial inclusion for the sake of inclusion. For example, policies promoting credit for all at all costs can lead to greater financial and economic instability. The report offers practical, evidence-based advice on policies that support healthy financial inclusion. Accompanying the report is an updated and expanded version of the Global Financial Development Database, which tracks Melinda Gates Foundation—filled this financial systems in more than 200 economies since 1960. The database—together with relevant research papers and other Financial Inclusion Financial Inclusion SKU 210361 background materials—is available on the report’s interactive website, www.worldbank.org/financialdevelopment. 2014 data gap by providing the first public set of indicators that consistently Financial Inclusion A Measured Approach to Ending measures people’s use of financial Poverty and Boosting Shared products in 148 countries around Prosperity: Concepts, Data, and the the world. The data and research Twin Goals (2014) have been cited in presentations, webinars, and research papers by ISBN 978-0-8213-9985-9 In 2013, the World Bank adopted two SKU 19985 the G20 Global Partnership for new goals to guide its work: ending GFDR_COVER_2014.indd 1 10/23/13 3:12 PM Financial Inclusion, AFI Global, extreme poverty and boosting shared Global Financial Development African Development Bank, the prosperity. The 2014 Policy Research Report 2014: Financial Inclusion Microfinance India Summit, the Report A Measured Approach to (2013) Inter-American Development Bank, Ending Poverty and Boosting Shared McKinsey & Company, CitiGroup, Prosperity: Concepts, Data, and the MIX Microfinance, the Reserve Bank Twin Goals synthesizes a wide array of of India, and Pew Charitable Trust, DECRG research products that both among others. Furthermore, four of informed the development of these the five “Basic Set” indicators of the goals and help to assess the likelihood G20 Global Partnership for Financial of reaching them. The list of products is Inclusion use Global Findex data. long, but best starts with PovcalNet— the public database of poverty data 20 I Research at Work 2015: Turning Insight into Impact Box 2.3: Policy with the $2 billion portfolio of World The CCT Policy Research Report Reports: Translating Bank-assisted livelihoods projects in has been highly influential. It Research Insights into India. The report has also received praise from top experts in the field served as the basis for World Bank engagement with governments in Development Impact of development. Nobel laureate many of the parts of the world in continued Roger Myerson has argued that which CCTs were not yet prevalent— “there are fundamental reasons in particular Africa and Asia. It to see the World Bank’s report launched critical discussions that Localizing Development, by Ghazala fed into the second wave of CCT Mansuri and Vijayendra Rao, as one programs around the world, in of the most important books on particular the role of cash versus development in recent years.”iv conditions, the complementarity between the role of service quality in translating increased utilization into outcomes, and the role of CCTs in mitigating the impact of shocks. And it was a pilot in the way that Public Disclosure Authorized A W O R L D B A N K P O L I C Y R E S E A R C H R E P O R T 47603 the World Bank’s impact evaluation agenda is now structured—namely around key clusters of priority areas. Public Disclosure Authorized To find out more about the Policy Localizing Development: Does Research Reports or the Global Public Disclosure Authorized Participation Work? (2012) Financial Development Reports and The World Bank has allocated over Conditional download publications in the series, visit the following links: http://econ. Cash Transfers Public Disclosure Authorized $85 billion to local participatory worldbank.org/prr and http://econ. Reducing Present and Future Poverty projects over the last decade. worldbank.org/gfdr. Moreover, participatory development has been a central focus of almost all Conditional Cash Transfers: major development donors, making it Reducing Present and Future one of the most important modalities Poverty (2009) of development assistance. While the topic has been the subject of a The Policy Research Report lot of controversy and debate, it has Conditional Cash Transfers: Reducing not received the analytic attention it Present and Future Poverty took deserves. stock of what is now thought of as the first large-scale wave of conditional The Policy Research Report on cash transfer (CCT) programs Localizing Development: Does around the world. At the time of its Participation Work? filled this gap writing in 2008, 29 countries had by bringing analytic insight and relatively large-scale CCT programs evidence to bear on this important underway; whereas 10 years earlier subject. It developed the conceptual only 3 countries did. CCTs were often foundations for participatory accompanied by impact evaluations, development; reviewed evidence and the report systematically on donors’ attempts to induce reviewed over 13 rigorous impact participatory development; and evaluations of those programs. laid out a policy proposal in which The report documented consistent participatory development programs reductions in poverty and increases take a “sandwich” approach where in consumption, school participation, bottom-up development is supported and the utilization of health services. by top-down efforts. However, it also showed that impacts on “final outcomes” such as learning The findings of the report have as measured by test scores, and been put into practice in World health or nutrition status of children Bank operations via the Social were hard to find. Observatory, which works closely Research at Work 2015: Turning Insight into Impact I 21 Notes To see listings of all of DECRG’s outputs, visit http://econ.worldbank.org/ 1. publications. RePEc, accessed November 14, 2014, http://www.repec.org. 2. To rank institutions in a field, RePEc generates a score for each institution 3. determined by a weighted sum of all authors affiliated with the respective institutions across 32 metrics. Rankings for each metric are then averaged using a harmonic mean after the highest and lowest scoring metrics for each institution are excluded. For a full discussion of RePEc’s ranking methodology, see Zimmermann (2012). http://home.uchicago.edu/~rmyerson/research/locgov.pdf, accessed 4. March 4, 2015.   22 I Research at Work 2015: Turning Insight into Impact Section 3: Data and Tools G ood research provides the theoretical and practical basis for creating and organizing useful datasets. Over the years, DECRG researchers have collected data from scratch if interesting policy questions could not be answered using existing data, with collection efforts spanning almost the full range of World Bank themes. In addition to enabling the exploration of key policy questions that could not be answered with the datasets available at the time, many of DECRG’s data collection efforts have made it possible to compare countries and track progress over time. Box 3.1 provides examples of how these efforts make a difference for policy when data is credible and constructed with foresight. Moreover, as analytical and computational tools used in development policy analysis have become more sophisticated, DECRG researchers have been at the forefront of making such tools widely available, thereby enabling and facilitating high-quality, policy-oriented analytical work by researchers around the world. Research at Work 2015: Turning Insight into Impact I 23 Box 3.1: Delivering Monitoring and Measuring in Brussels, and the US House Ways and Means Committee. Every Protection and Its Implications Impact through Data international negotiation—from the DECRG’s new public databases Trans-Pacific Trade Partnership to and innovative techniques to ASEAN—is using DECRG’s research measure trade restrictions and and data products as key resources. their consequences have helped restrain protection, advance policy For more on DECRG’s public reform, and inform international databases, visit http://econ. trade negotiations. DECRG worldbank.org/tradedata identified relatively early the need for more effective monitoring and Worldwide Governance Indicators evaluation of policies affecting the exports and imports of developing Governance and institutions are countries. Many of the tools that central to understanding why were developed were adopted as some countries are poor and part of the heightened trade policy others rich and what can be done surveillance during the recent to achieve inclusive growth. The financial crisis, and have continued to Worldwide Governance Indicators be used in the post-crisis period. (WGI) project has made it possible to systematically evaluate the The Temporary Barriers to Trade relationship of governance to Database and accompanying research development by supplying aggregate revealed use of opaque instruments and individual governance indicators of protection like anti-dumping duties for 215 economies over the period and emerged as a critical component 1996–2013. The WGI are a set of six of global monitoring efforts. The composite indicators of governance Overall Trade Restrictiveness Index measuring Voice and Accountability, made it possible to combine tariff Political Stability and Absence of and non-tariff measures into a single, Violence, Government Effectiveness, transparent measure of protection Regulatory Quality, Rule of Law, and in goods trade. The Services Trade Control of Corruption. The data Restrictions Database provided a website registers an average of first global view of applied policy in 200,000 visits per year. The WGI have this previously overlooked area. The become very widely used among real-time information provided by the academics, policy makers, and in Bank’s trade policy monitoring work the private sector. Most notably, the was disseminated not only through WGI are used by the US Millennium traditional research outlets, but also Challenge Corporation for its cross- through policy briefs, op-eds and country aid allocation system. Users citations in prominent media outlets of these indicators in the private such as the Economist, Financial sector include the major commercial Times, The Wall Street Journal, risk rating agencies, as well as the The Washington Post, Reuters, Disney Corporation’s corporate social BusinessWeek, Xinhua, and CNN. responsibility team. Researchers from DECRG have been invited to present findings to the For more on WGI, visit http://econ. WTO Council, the European Union’s worldbank.org/wgi. Member States Committee on Trade 24 I Research at Work 2015: Turning Insight into Impact Data Compilation Agreement is widespread that thoughtful data collection, guided by researchers who understand what types of information are most needed to address fundamental policy questions, is an important World Bank contribution to the field of development. The research-based data products generated by DECRG fall into three categories. Some are compilations of existing data series, while others are collections of primary data. The third category comprises datasets that are constructed by piecing together information from different sources and then performing some modeling. A vital characteristic of all three types of datasets is that they are valuable public goods: the same data can be used by many researchers and policy analysts to answer a broad range of development policy questions. Table 3.1: Compiled Databases Financial Development For a broad set of countries since 1960, the Financial Development and Structure and Structure Database Database provides statistics on the size, activity, efficiency, and stability of banks, equity markets, and bond markets. Database of Political The world’s most comprehensive compilation of data on political institutions, this Institutions database consists of annual data on 100 variables for 100 countries over the period 1975–2010. Worldwide Governance This database reports the perceptions of and experience with governance of survey Indicators respondents and experts in the public and private sectors worldwide. Data Collection Data compilation relies on data being there to start with. When World Bank President Robert McNamara saw so many gaps in the poverty and inequality data series in the 1979 World Development Report, the reason was not that his staff had not bothered to compile the data needed for McNamara’s reorientation of the Bank toward poverty reduction; the data simply did not exist. Most countries had never fielded a household survey, and those that had had not asked the necessary questions to accurately estimate a household’s consumption. To address this deficiency, the Living Standards Measurement Study (LSMS) team set about designing a consumption questionnaire that could be used in household surveys in different settings, and they fielded it in multiple countries. Other organizations subsequently embraced the philosophy of the consumption module, thanks in part to a three-part volume the World Bank produced explaining the idea. Building on this legacy, World Bank researchers continue to set the standard for data collection efforts in everything from financial inclusion to service delivery. Research at Work 2015: Turning Insight into Impact I 25 Table 3.2: Collected Databases Absenteeism Surveys Random unannounced surveys of schools and health facilities determine rates of staff absenteeism in an effort to discover why these institutions often fail to deliver results. Agricultural Trade Covering the period 1955–2007 and currently being updated, this database compiles Distortions Database detailed country-level information on taxes, tariffs, and other trade distortions in agriculture. Bank Regulation and A unique source of comparable world-wide data on how banks are regulated and Supervision Database supervised around the world, this database provides information on bank regulation and supervision for 143 jurisdictions. Enterprise Surveys Enterprise Surveys provide the world’s most comprehensive company-level data in emerging markets and developing economies. Data are now available for 135 economies. The success of the initiative led to the project being spun off into its own dedicated unit outside of DECRG. Global Financial The new Global Financial Inclusion Indicators gather data on access to financial services Inclusion Indicators and the financial inclusion of individuals around the world. Learning and Learning and Educational Achievement in Pakistan Schools (LEAPS) gathers data both on Educational the distribution of schools throughout Pakistan and on the quality of education provided Achievement in Pakistan by a representative sample of over 800 public and private schools in rural Punjab. Living Standards The Living Standards Measurement Study has fielded multi-dimensional household Measurement Study surveys in many countries, collecting data on consumption, income, and various other indicators that help in assessing household welfare and living conditions. Maximum Residue The Maximum Residue Level Database collects maximum pesticide residue levels for Level Database agricultural products in a number of developed and developing countries. Public Expenditure Public expenditure tracking surveys track government spending as it flows from the Tracking Surveys finance ministry down to grass-roots schools and health facilities. Service Delivery A new Africa-wide initiative that tracks service delivery in education and health across Indicators countries and over time, this project collects nationally representative data that focus mainly on quality of service delivery in primary schools and at frontline health facilities. Services Policy The Services Policy Restrictiveness Database contains information on policy and Restrictiveness regulatory measures affecting international trade in services for a total of 102 countries, Database of which 78 are developing countries. Data Construction The Database of Political Institutions and the Worldwide Governance Indicators lie at one extreme of database creation, and LSMS and Enterprise Surveys lie at the other. The former are highly valued compilations of existing data series, while the latter are collected through questionnaires and observation in thousands of households and firms all over the developing world. Between these two extremes lies another type of dataset: one that is constructed by piecing together information from different sources and then performing some modeling. 26 I Research at Work 2015: Turning Insight into Impact Table 3.3: Constructed Databases Education Attainment Using household surveys, this database constructs indicators of levels and and Enrollment Profiles inequalities in schooling attainment and participation. The data are periodically Database updated, and currently cover over 100 countries with on average three time periods per country. Environmental Data DECRG’s environmental datasets are constructed by piecing together information from different sources, including information gleaned from maps and satellite images, and then performing some modeling to address various questions about the environment. Exporter Dynamics This database provides statistics on export growth and dynamics in the agriculture, Database mining, and manufacturing sectors, and is constructed using exporter-level customs transaction data as an input for 30+ developing countries and a handful of developed countries. Global Bilateral Migration The Global Bilateral Migration Database consists of five 226-by-226 bilateral Database matrices of migration stocks by gender for each decade over the period 1960–2000. I2D2 A constructed “international income distribution database” assembling unit record survey data on incomes, consumption, and a wide range of ancillary variables. Legal Data Legal Data provides information on court decisions relating to public interest litigation by type of claim and social class of the claimant. Overall Trade The Overall Trade Restrictiveness Indices summarizes the trade policy stance of Restrictiveness Indices developing and developed countries by taking into account the structure of trade policy (both tariff and nontariff barriers), trade flows, and trade elasticities. Temporary Trade Barriers An outgrowth of the Global Antidumping Database, the Temporary Trade Barriers Database Database collects and freely disseminates detailed data on temporary trade barriers permissible in the World Trade Organization system. Tools The range of computational tools produced by DECRG are designed to empower researchers and policy practitioners in developing countries to conduct their own research with minimal technical assistance. One case in point is Survey Solutions, which was developed to help national statistical organizations in developing countries by reducing the costs and improving the timeliness and quality of survey data. Since the release of the first version at the end of 2013 more than 30 countries have used Survey Solutions for data collection, including the National Sample Survey Office of India; Bangladesh Bureau of Statistics; Statistics South Africa; RosStat of Russia; and the Brazil Institute of Geography and Statistics. Survey Solutions has a wide presence in Africa with surveys currently being conducted in Niger, Cote D’Ivoire, South Sudan, Tanzania, and South Africa. Recent labor force surveys in Kuwait and Bhutan and an impact evaluation in the war zone of Peshawar, Pakistan conducted with minimal technical assistance from the World Bank demonstrated the flexibility and ease of use of Survey Solutions even in the most difficult environments. Research at Work 2015: Turning Insight into Impact I 27 To promote the adoption of methodological best practices and new tools, DECRG emphasizes building capacity across partner countries. Capacity building includes a combination of sustained hands-on training through long-term engagement and country presence, and more formal classroom as well as web-based and remote training. The assistance is delivered in partnership with other institutions and as part of broader global initiatives like the Global Strategy to Improve Agricultural and Rural Statistics and the National Strategy for the Development of Statistics, contributing to the sustainability of the work of LSMS. The success of this model can be seen with increasing government contributions to the maintenance and expansion of systems, expanding support by a large number of institutional partners, growing use of the information collected, and decreasing need over the years for technical assistance. Table 3.4: Tools ADePT A platform for automated economic analysis, ADePT software is widely used for policy analysis in governments, think tanks, and universities around the world. The ADePT project has generated five book publications; training seminars are conducted in all regions of the world. PovCalNet This software allows users to replicate the Bank’s official global, regional, and internationally comparable country level poverty estimates as well as the shared prosperity indices. PovMap PovMap lets users produce small area estimations of poverty, inequality, nutrition status, and other indicators. Survey A software system that relies on tablet PCs to capture data and on cloud computing for data Solutions aggregation and survey management, Survey Solutions reduces the time lag between data collection and data analysis and dramatically improves data quality. To find out more about the data and tools supported by DECRG, visit: http://econ.worldbank.org/data   b 28 I Research at Work 2015: Turning Insight into Impact 20 18 Section 4: 16 Evaluative 14 12 10 08 06 04 Research 60 7 0 2 80 005 PPP $) 90 100 5 0 o n t h ( rm 02 3 0 40 m e p e r capita pe co 0 10 20 e x p e n d iture or in tion 0 Consump c. 2030 M 20 easuring the causal links between policies 18 and outcomes is fundamental for good policy 16 making. Decisions based on weak inferences may 14 result in ineffective or even damaging policies. Take, 12 for example, the case of a government that reacts 10 to an economic recession by tightening immigration 08 policies—the slow-down in migration flows might be 0 100 06 attributed to enforcement efforts, even though 9 the 0reducing 80 04 recession itself may 6 be directly 7 the$pull) on 5 0 0 o n t h ( 2 005 PPP 02 migrants. 40 This type r cap of per m itaerror in evaluating the effect of 0 p e 0 2 0 3 d itpolicy u r e or in often m e co goes unnoticed. e x c lu d in g China) n ( 10 tion expe India and Pacif aribbean ic 0 Consump Ea s t A s ia C a t in A m e rica and Ch in When Africa India) done well, L impact a evaluations can deliver precise a h a r a n Sub-S ia (excluding estimates of the causal relationship between a policy o andr in come dis So u t h A s p t io n outcomes by comparing treatment - s pand e c ific c o n s um control groups a r s a nd e x t end c ou n t r y s ur v e y y e d a t a . before anddafter v a r ia a n c e o f t he policy intervention. o la t e d b e t we e n This approach u n to d e r t h e a s s ump et an e meaninference r ly inter p jec tio n o r ld B a n k PovcalN ased on th causal a r e lo g - li has ne a usefully expanded 3 0 s h o w s a proeconomist’s the o n s t a n t in equalit y alysis of W mal distr ibutions, b ar s, sur v ey m e an toolbox s by allowing h . T h e g aph for 20 to rresearchers 0 0 0 t o establish an d c 2011 well-defined n o r y e w t 2 s sume log te annual reference ld consucounterfactuals mption gro ates o v e againstr t he p er iod which a policy can be evaluated. e r a h o r ar. To gen 1, using real house r ical aver age grow th 01 to Establishing the direction and magnitude of causal r ward to 2 d 2030 at their his it y. a n a r een 2011 pur chasing power p relationships can guide policy makers and donors in = year. PPP identifying the right policies to achieve a desired objective, calculating the cost-effectiveness of various policy options, and justifying a response or a scale-up. Research at Work 2015: Turning Insight into Impact I 29 Working hand-in-hand with operations, DECRG has played a central role in expanding the use of impact evaluations in World Bank projects, enhancing rigor and quality, and drawing global lessons from this body of work. The publication of the 2009 Policy Research Report Conditional Cash Transfers: Reducing Present and Future Poverty set a precedent for the way that the World Bank’s impact evaluation agenda is now structured around key clusters of priority areas (see Box 2.3). A few years later, the Independent Evaluation Group (IEG) also specifically cited the role of DIME—a unit within DECRG with a mandate to produce high-quality impact evaluations—in improving quality control in impact evaluations (IEG, 2012). And DECRG’s publication record demonstrates its outsized role in turning the results of impact evaluations into global knowledge: of the 35 papers published in 2014 as part of the World Bank Impact Evaluation working papers series, 28 were produced either jointly or solely by DECRG.i The World Bank is the largest producer of impact evaluations among development institutions, with the stock of impact evaluations rising from less than 20 in 2003 to around 193 in 2014. DECRG is closely involved with a large fraction of these. DIME’s portfolio alone contains 175 completed or ongoing impact evaluations (see below for further information). And DECRG’s engagement in impact evaluations cuts across all sectors of the World Bank, with projects that range from boosting the productivity of the private sector (Box 4.1) to assessing results-based financing in healthcare (Box 4.2). reduced inventory, and within Box 4.1: Better Can differences in management three years led to the opening of practices across firms explain Management Practices differences in productivity, especially more production plants (Bloom et for a More Productive in developing countries where these al., 2013). These findings have been Private Sector gaps appear particularly large? To influential in better understanding investigate this, a team of researchers the role of management practices in from DECRG and Harvard University explaining performance differences ran a management field experiment across firms. They have also grabbed on large Indian textile firms. The the attention of policy makers: team provided free consulting on the government of Colombia has been influenced by this research to tr ibutions management practices to randomly chosen treatment plants and conduct a larger-scale program for de d b ac k - compared their performance to a improving management in the auto parts industry, and is evaluating the ption that set of control plants. They found that adopting these management individualized consulting approach since the practices raised productivity by 17 used in this research alongside a novel group-based approach percent in the first year through improved quality and efficiency and designed by a World Bank team. 30 I Research at Work 2015: Turning Insight into Impact Initial evidence generated from Box 4.2: Assessing Established in 2007, the Health these evaluations has shown that Results Innovation Trust Fund (HRITF) Results-Based has a mandate to support results- paying for results and empowering Financing in Healthcare based financing approaches in the health workers and managers at health sector. The HRITF aims to the frontlines can improve access improve maternal and child health to quality health services and save around the world, and has committed lives. Evaluations have found that $420 million towards reaching the results-based financing helped Millennium Development Goals to boost HIV testing rates in Rwanda, reduce child mortality and improve especially among couples; increased maternal health. the likelihood that pregnant women delivered in a facility and received The HRITF takes a programmatic post-natal care in Zimbabwe; and approach to developing the evidence drastically cut the rate of in-hospital base for implementing successful neonatal mortality of babies in results-based financing mechanisms. Argentina. Damien de Walque, a DECRG Senior Economist, coordinates a team of For more information, visit: http:// researchers who oversee the quality www.hritfreport.org/ of the entire World Bank portfolio of HRITF impact evaluations, which includes 33 on-going evaluations as of Fiscal Year 2014. Development Impact Evaluation (DIME) Established in 2005, the Development Impact Evaluation (DIME) team’s mandate is to generate high-quality operationally relevant impact evaluation research to transform development policy. It works closely with World Bank operations to increase the use of impact evaluation in the design and implementation of public policy, improve the quality of World Bank projects, strengthen country institutions for evidence-based policy making, and generate knowledge in strategic development areas. DIME achieves these through (i) designing evaluations to inform decisions in collaboration with government partners, (ii) building counterpart agencies’ capacity to do this systematically, and (iii) drawing lessons and sharing them face-to-face with global audiences. Research at Work 2015: Turning Insight into Impact I 31 A critical feature of DIME’s approach to impact evaluation is the focus on both the question of “what works?” and “why/how it works?” DIME impact evaluations both evaluate the effectiveness of packages of interventions (the “what”) and experiment with mechanisms (the “why/how”) to better understand what drives impact. Examining both of these questions allows DIME to provide actionable evidence to its implementing partners. And to ensure evidence is turned into action, the DIME model transfers the knowledge and tools needed to support evidence-based policy making to country institutions through a learning-by-doing approach (Box 4.3). DIME’s 175 projects span nearly all of the World Bank’s Global Practices (Figure 4.1). The largest concentration of projects fall under the Agriculture Global Practice (20 percent), with substantial shares in Health, Nutrition, and Population (15 percent), Social, Urban, Rural, & Resilience (13 percent), and Education (11 percent). A number of new initiatives are helping to broaden DIME’s coverage even further, including the ieGovern Initiative that is pushing the frontier in terms of what works in governance reform, the Evidence for Peace designed to fill the knowledge vacuum about what works in settings characterized by fragility, conflict, and violence, and Impact Evaluation to Development Impact (i2i), a World Bank fund supported by the Department for International Development (DFID) that will increase the use of impact evaluations in areas that have traditionally been under-evaluated.ii Agriculture Figure 4.1: DIME Impact Health, Nutrition, & Population Evaluations by World Bank Social, Urban, Rural, & Resilience Global Practice Education Trade & Competitiveness Governance Finance & Markets Transport & ICT Water Energy & Extractives Social Protection & Labor Environment & Natural Resources 0% 5% 10% 15% 20% 32 I Research at Work 2015: Turning Insight into Impact tested the impact of a pilot that Box 4.3: The Learning- In 2009, the government of Brazil introduced new books, teacher approached the World Bank with by-Doing Approach a request to provide a credible training, and school incentives to in Brazil: Piloting a and independent evaluation of deliver a comprehensive high-quality High School Financial their proposed financial education financial education. Literacy Program strategy, the Estratégia Nacional The evaluation showed that the de Educação Financeira (ENEF). However, existing evidence on program was highly effective at financial literacy programs around improving financial knowledge, the world pointed to very little (if any) attitudes, and behavior, and impact on program recipients. Thus, also highlighted the underlying before the government could move mechanisms that led to these from a national policy to a tangible improved outcomes (Bruhn et al. national program, there was a need 2013). This impact evaluation also to evaluate the flagship school-based showed that, despite students’ financial literacy curriculum and socio-economic background and provide an evidence base for any the quality of schools, an intensive national scale up plans. and curricula-integrated financial literacy program can impact students’ DIME collaborated with the Ministry financial outcomes with potential of Education, six State Secretaries, long-run implications on individuals’ the financial regulatory board, the decision-making. The Ministry of Central Bank, the Brazilian stock Education decided to scale up market, and other financial regulators the program nationally, seeking to to run the first and only randomized improve students’ long-run outcomes evaluation of a financial education through changes in their financial program in schools encompassing behavior and knowledge. The scale- nearly 26,000 students in 900 up is currently under implementation. schools across Brazil. The evaluation Research at Work 2015: Turning Insight into Impact I 33 Notes 1. To see the full collection of Impact Evaluation Policy Research Working Papers, visit: http://documents.worldbank.org/curated/en/docsearch/ collection-title/Impact%20Evaluation%20series. 2. For more on the work of DIME, see: http://www.worldbank.org/dime.   34 I Research at Work 2015: Turning Insight into Impact DECRG Alumni: Where Are They Now? DECRG has an impact not only through its own outputs, but also through the human capital and networks gained by the staff who pass through its doors. Many of the researchers who work in DECRG have gone on to take leadership positions elsewhere, whether within the World Bank or in senior positions outside of the institution. What follows is a selection of DECRG alumni, along with positions they have held since leaving DECRG. World Bank Shantayanan Devarajan William Maloney Chief Economist, Middle East and North Africa Region, Chief Economist, Trade and Competitiveness Global World Bank Practice, World Bank Francisco Ferreira Martin Rama Chief Economist, Sub-Saharan Africa Region, World Bank Chief Economist, South Asia Region, World Bank Mary Hallward-Driemeier Ritva Reinikka Senior Principal Specialist, Jobs Cross Cutting Solution Director (retired), Human Development, Africa Area, World Bank Region, World Bank Elizabeth King Former Director of Education, Human Development Network External Thorsten Beck Philip Keefer Professor of Banking and Finance, Cass Business School in Principal Economic Advisor, Institutions for London Development, Inter-American Development Bank Gerard Caprio Peter Lanjouw William Brough Professor of Economics, Williams College Professor of Economics, VU University Amsterdam Paul Collier Ross Levine Professor of Economics and Public Policy, University of Willis H. Booth Professor of Banking and Finance, Oxford & Director, Centre for the Study of African Economies University of California Berkeley David Dollar William Martin Senior Fellow, Brookings Institution Senior Research Fellow, IFPRI William Easterly Branko Milanovic Professor of Economics, New York University Senior Scholar, Luxembourg Income Study Center, City University of New York Caroline Freund Lant Pritchett Former Chief Economist, Middle East and North Africa Professor of International Development, Kennedy Region, World Bank & Senior Fellow, Peterson Institute School of Government Research at Work 2015: Turning Insight into Impact I 35 External continued Ahmed Galal Claudio Raddatz Managing Director, Economic Research Forum, & Former Division Director of Financial Policy, Central Bank of Finance Minister, Egypt Chile Jeffrey Hammer Martin Ravallion Professor of Economic Development, Princeton University Edmond D. Villani Professor of Economics, Georgetown University Bernard Hoekman Norbert Schady Research Director of Global Economics, European University Principal Economic Adviser, Social Sector, Inter- Institute American Development Bank Patrick Honohan Klaus Schmidt-Hebbel Governor, Central Bank of Ireland Former Chief Economist, OECD Beata Javorcik Emiliana Vegas Professor of Economics, Oxford University Chief, Education Division, Inter-American Development Bank Emmanuel Jimenez Alan Winters Executive Director, 3ie (International Initiative for Impact Former Chief Economist, DFID & Professor of Evaluation) Economics, University of Sussex 36 I Research at Work 2015: Turning Insight into Impact Research Leadership Team The Research Director along with the Research Managers and Senior Advisers are responsible for providing DECRG with intellectual leadership. For a complete list of staff by Global Practice, visit: http://econ.worldbank.org/staff. Director, Managers, and Senior Advisers Asli Demirguc-Kunt Maria Soledad Martinez Peria Director of Research Manager, Finance and Private Sector Development Aart Kraay Aaditya Mattoo Senior Adviser Manager, Trade and International Integration Arianna Legovini Luis Serven Manager, Development Impact Evaluation Senior Adviser, Macroeconomics and Growth Michael Lokshin Michael Toman Co-Manager, Surveys and Methods Manager, Environment and Energy Calogero Carletto Adam Wagstaff Co-Manager, Surveys and Methods Manager, Human Development and Public Services Francisco Ferreira Senior Adviser, Poverty and Inequality Research at Work 2015: Turning Insight into Impact I 37 References Bloom, Nick, Benn Eifert, Aprajit Mahajan, David McKenzie, and John Roberts. 2013. “Does Management Matter? Evidence from India.” Quarterly Journal of Economics 128 (1): 1-51. Bruhn, Miriam, Luciana de Souza Leão, Arianna Legovini, Rogelio Marchetti, and Bilal Zia. 2013. “The Impact of High School Financial Education: Experimental Evidence from Brazil.” Policy Research Working Paper 6723, World Bank, Washington, DC. Das, Jishnu, Quy-Toan Do, Karen Shaines, and Sowmya Srikant. 2013. “U.S. and Them: The Geography of Academic Research.” Journal of Development Economics 105: 112-130. Doemeland, Doerte, and James Trevino. 2014. “Which World Bank reports are widely read?” Policy Research Working Paper 6851, World Bank, Washington, DC. Engemann, Kristie M., and Howard J. Wall. 2009. “A Journal Ranking for the Ambitious Economist.” Federal Reserve Bank of St. Louis Review 91 (3): 127-139. Independent Evaluation Group. 2012. World Bank Group Impact Evaluations: Relevance and Effectiveness. World Bank, Washington, DC. World Bank. 2007. Finance for All? Policies and Pitfalls in Expanding Access. Policy Research Report. Washington, DC: World Bank. ———. 2009. Conditional Cash Transfers: Reducing Present and Future Poverty. Policy Research Report. Washington, DC: World Bank. ———. 2012a. Global Financial Development Report 2013: Rethinking the Role of the State in Finance. Washington, DC: World Bank. ———. 2012b. Localizing Development: Does Participation Work?. Washington, DC: World Bank. ———. 2012c. Research at Work: Assessing the Influence of Bank Research. Washington, DC: World Bank. ———. 2013. Global Financial Development Report 2014: Financial Inclusion. Washington, DC: World Bank. ———. 2015. A Measured Approach to Ending Poverty and Boosting Shared Prosperity: Concepts, Data, and the Twin Goals. Policy Research Report. Washington, DC: World Bank. Zimmermann, Christian. 2012. “Academic Rankings with RePEc.” Federal Reserve Bank of St. Louis Working Paper 2012-023A, St. Louis. Research at Work 2015: Turning Insight into Impact I 39 I ES AND ? POLIC i fro e co d he th o as an ea e to le t me rm a n e m si do ev op of re t t wi its p ab sess d fi n th als ot m ere pr o c nt s a th an m th m th en e c r o s s e c on omies e s a ic Ratio of d v . it ser v na ak nte we ns nd de p o s e o tio W rta . T er, H n a em . H lu a TS he lo dit private cre p l ng s o world development report GDP per IG im odi st nk econ.worldbank.org/research p b flo -co or ). R n Ba io to GDP th il t o low ld p/ a Deposit nt capita, J ve s. n n f ri e n As m 99 ar f (average 19 re ee te the ev o ye accounts a n fr © ict e in a sk 2003 ($) ep Loan (a s ate and rh re 90 Deposit- is o Fa u a iv rty to 2003) n le uti iti wn ct ha per capita ove rty accounts f fe p et io m o income rat d ve pr e s uld kn m on g an Loan- po rty or 1,933 fe s s of ver low pove m (number) on tiv wo ing eo o per capita c t, gh ty or es ver — m ff ms nt i io nc for st, h h po rty income rat 3,381 sf co st-e ste me ob o l h an se g ore hig ve 2.75 (number) s a co sy ple h f est, w po le 0.205 b ea Or g 161.25 915 ra and age im i H fo t, l r o m ec w 0.58 s th w s Lo fore 15.41 llu ed rai tie Ne 0.076 in 368.73 31,202 w a e i ifi d cul o in L t 4.42 da n 1.00 s i c ide ge iffi e ve no 1.77 1.025 ag ba ell- log 1 D 111.38 376 l m se as nt d 154.19 da 0.26 c . m te M ch ing 2 1.93 0.245 e d 3,119.95 29,205 su an t o st h ea 41.23 tt 1.60 pr Cl h o 1.84 a 0.773 es pe 228.75 894 P id um w e 647.64 re w 5.22 0.38 0.558 in es 3,080.31 1,682 d ris 54.73 te 5.81 ul C 21.09 — es 40.63 1 sr 59.47 2. 1.87 ha 27.89 P 429.40 2,788 g a in 9.53 M od 3.19 0.346 Flo 114.09 2,538 0.40 0.149 630.86 4,591 6.18 0.26 0.694 49.59 1,351.37 1,747 4.24 0.46 0.262 73.85 1,044.82 8,375 1.6 0.42 0.424 417.74 612.21 0.42 39,429 n — 1.100 si 1,922.83 1,821 ea ar — 0.22 e — 0.335 lu B 2,706.07 s. — e 0.10 ve 1.0 2.09 0.8 le 719.52 2,066 e th 0.8 450.9 9 0.7 0 n ,2 6.71 i 0.353 ea 6. es e 200 2,204 sted ch 0.6 .6 5 0.10 0.63 te ood br re ta 0.5 da s 0.047 efo ica -W 0 419.54 2,696 th o ti nd ir w nd 0.12 s i Mu 0.4 rate sh tric he or- 2.63 0.322 ata ro nd s s t f 456.69 ge di to n 29,267 y 0.4 ert ar si a 77.09 am 0.3 Pov pa sub nal r tio icts ir 1.13 w ed ros 0.39 o o r e 0.857 kec tw ch tio p st h 444.42 16,203 te e: G 0.2 e ea por pro bdi on t r 126.89 of 2 s No urc u 0.40 . e A e e 0 th el pro th t. S sed e o th 4.75 :R 0.546 on So 0.1 on pan e is ows tric , ba bov aps s . 1,800.84 2,009 orti 67.09 s h In siz sh dis ies d a m w t 0.0 0.0 n B ho ric 0.29 p ap ). b r — 0.189 Pro gr rict bble raph su ego er a nel l C s dist — nd ist bu g ch at ov Pa ne ub a. s a bd e he ea d c t c e. Pa s re 2,417.64 965 0.55 MIND, SOCIETY, 0.83 ap (su . Th n. T e of ode res rat es. ach ct a 3 10 — 1,001 776.48 h m e tan bble latio rat or-c w fo er ty ori for istri : T a u u ty ol lo ov te s bd g 403.54 e 1.37 3.19 te am a b pop er r c us ) p ca ate su 0.388 2,061 45.79 No kec by t’s pov fou ers esia se ion r tal 571.03 AND BEHAVIOR— by own tric and into gh v don the tat /to — sh bdis ver fied f hi r In g to ores los u s t c o s i o f s ion e ( rd d e o 6.13 i n e f s t s 287.27 0.74 0.04 0.281 16,686 es e cla nat rag cco and for 0.859 25,429 67.27 a m i ave ts a tes d a r b s 2,249.28 — co low tric ty ra fine be bdis ver s de 2.91 — 0.750 1,858 48.19 su e po te i 1.58 0.47 th e ra 975.64 0.721 434 709.90 Th 2.35 1.41 0.258 328.15 465.48 4,224 8.2 6.26 — 80.39 69.98 5 5,273