68788 Forthcoming in Review of International Organizations The World Bank‘s Publication Record Martin Ravallion and Adam Wagstaff1 Development Research Group, World Bank 1818 H Street NW, Washington DC, USA October 2011 Abstract: The World Bank claims to be a ―knowledge bank,‖ but do its knowledge products influence development thinking, or is the Bank merely a proselytizer? The World Bank is a prolific publisher; for example, it has published more journal articles in economics than any university except Harvard. But what about their impact on development thinking? Using citation data from Google Scholar it is hard to discern more than a negligible impact for a great many Bank publications. However, a sizeable minority of its journal articles and books have been highly cited. Compared to leading research universities and other international institutions, the Bank‘s ranking in terms of widely-used citation-based indices is no lower than for is journal article counts. This suggests that the Bank‘s research does much more than proselytize. Keywords: Bibliometrics, World Bank, citations, h-index, journals 1 We owe an enormous debt of gratitude to Qinghua Zhao; without his programming skills this paper would have been substantially less interesting. We are also grateful to Imran Hafiz for help in retrieving data from online databases, and to Trinie Angeles for assembling the database of research staff. We are grateful for helpful comments to this journal‘s editor, two anonymous referees, Angus Deaton, Asli Demirguc -Kunt, Francisco Ferreira, Deon Filmer, Bernard Hoekman, Martin Rama, Lyn Squire, Dominique van de Walle, Colin Xu, and Jane Zhang. The views expressed here are those of the authors and need not reflect those of the World Bank, its Executive Directors, or the countries they represent. 2 I. INTRODUCTION The World Bank (hereafter ‗the Bank‘) uses three main sets of instruments to help the developing world reduce poverty: lending; policy advice and technical assistance; and the generation, synthesis and dissemination of data and knowledge. The first two are targeted at specific countries, while the third is a mix of country-specific and global initiatives. Much of the Bank‘s recent reform efforts have focused on the third instrument, knowledge. Those efforts have aimed to assure that lending operations and policy advice are underpinned by solid evidence and that the results are properly evaluated. Their aim is to give all countries access to data, tools and knowledge that will help them craft successful policies and strategies, whether or not the Bank is involved in financing their implementation. This is consistent with the view that, as a global public good, knowledge on development would be undersupplied without the efforts of a global institution like the Bank (cf. e.g., Gilbert et al. 1999). As the Bank‘s current president, Robert Zoellick, puts it: ―Our financial resources are significant – but they are finite. By contrast, knowledge is potentially unlimited: the more it is shared, the more new ideas develop, and the more improvement is possible. When strands of knowledge are connected, the possibilities for increased prosperity and improvements in human welfare multiply.‖ (World Bank, 2011 p.v) Thus the Bank embarked in 2010 on an Open Knowledge initiative. Highlights include free access to the Bank‘s World Development Indicators, and an Open Knowledge Repository which will make available free of charge all Bank documents, reports, and publications. Of the Bank‘s three roles, it is its knowledge role where the evidence base is weakest. There is a fairly extensive analytic literature on the Bank‘s role as a lending institution. A number of econometric studies have analyzed Bank lending policies in general (Deininger et al. 1998; Wane 2004; Dollar and Levin 2005; Dollar and Levin 2006; Flores and Nooruddin 2009; Marchesi and Sirtori 2011; Knack et al., 2011), as well as evaluating specific projects (see, for example, van de Walle and Mu 2007; Chen et al. 2009; Wagstaff 2011). By contrast, analytic work – particularly of a comparative and quantitative nature – on the Bank‘s role as a knowledge institution has been very limited. 3 This paper tries to help fill this gap. Our goals are twofold: to take stock of an important part of the Bank‘s knowledge portfolio, namely formal publications; and to assess the influence of these publications on development thinking. Previous research on the Bank‘s role as a knowledge institution has been largely qualitative, and the evidence presented on its impact on development thinking has been rather anecdotal. Gavin and Rodrik (1995) argue that the Bank is ―not … at the forefront of economic ideas‖. It is, they say, ―difficult to pinpoint a single important idea or method in development economics that has its origin in the World Bank.‖ They conclude: ―Where the Bank‘s strength lies is in its tremendous powers to spread and popularize ideas that it latches on to.‖ The Bank, they conclude, is not a source of ideas, but rather a highly effective proselytizer. This assessment stands in marked contrast to the assessment by Squire (2000) – a former Director of the Bank‘s research department. He concludes, on the basis of a mix of bibliometric data and surveys of Bank staff and clients (the details of which are not presented in his paper), that ―Bank research is influential among policymakers, [and] well regarded by other researchers.‖ The Squire assessment is clearly less impressionistic than that of Gavin and Rodrik, but one is still left wondering ―compared to what?‖ We try to build on these studies, by using bibliometric data to provide a quantitative and comparative perspective on the Bank‘s influence on development thinking. We present in section III bibliometric data over time and by type of publication; and in section IV we compare the Bank to other institutions – both other international development agencies and top universities. We include in our definition of publications not just the Bank‘s own publication series, but also articles authored by Bank staff in independent peer-reviewed journals and books and book chapters not published by the Bank. This is a broader view of Bank publications than is typically adopted within the institution; indeed, journal articles and books published by commercial publishers are not seen as part of the Bank‘s publications portfolio. Of course, the Bank‘s knowledge portfolio is larger than just formal publications, even including journal articles. Most of the Bank‘s economic and sector work (ESW) never emerges as a formal publication. And the Bank‘s ‗knowledge management‘ work, which might take the form of a briefing note or a searchable database, rarely results in a formal publication. When a piece of 4 work does make it into a publication, the implication is that it has been considered by the publisher – whether the Bank or a commercial publisher – not only to be of a high standard (much of the Bank‘s ESW would pass this test), but also to be of potential interest to a large number of people working and writing on development. Some of these publications – particularly those published by the Bank itself – may synthesize existing knowledge, rather than add to new knowledge. In such cases ‗influence on development thinking‘ is probably the wrong yardstick against which to assess impact. These publications may have a considerable impact on policy and practice, by connecting existing knowledge to policymakers, their advisors, and voters. The World Development Report (WDR) (familiar to almost everyone working on development) is probably a good example of such a publication. But the Bank has aspirations to be a generator of new knowledge, not just as a proselytizer. It aims not just to reflect development thinking, but to shape it. The Bank has a dedicated research department, whose staff members publish in scholarly journals. And – as we show below – it also has a large number of staff outside the research department who publish in such journals. In these publications, and in many of their books and book chapters, Bank authors aim to contribute new knowledge. The new knowledge may not always be fundamental in a global sense, and it may only be applying existing techniques, developed by others, to a developing country or set of countries. But for these countries, if not the developing world as a whole, the work may represent a dramatic increase in the knowledge base. Often Bank authors aim to create knowledge in a more fundamental way as well, by developing new ideas and methodological tools, and presenting relevant empirical evidence. As with much research, the durable contribution to knowledge often comes from the idea rather than the accompanying empirical work, which often gets surpassed in time by more sophisticated analyses. In short, while ‗influence on development thinking‘ is not the only yardstick against which to assess the impact of the Bank publications, it is a legitimate and important one, given the Bank‘s aspirations. Citations provide our measure of the influence of the Bank‘s publications on development thinking. Our assumption is that a publication that is considered by others to 5 contribute to new knowledge will get cited, following normal scholarly practices. And the more knowledge it adds, the more often it is likely to get cited. Wherever possible we report citation data from Google Scholar (GS). GS allows us to extend the range of publications for which we obtain citation data: we can get citation data for articles in new journals, books and working papers, not just articles in established journals.2 GS also allows us to extend the scope of our assessment of influence on development thinking: we are able to look for citations by others not just in established journals, but also in books, technical reports, new journals, and even dissertations. GS is not problem-free as a source of citation data; in the next section we note its limitations. II. DATA AND METHODS We construct two sets of databases, corresponding to the two types of comparisons we make: comparisons within the Bank‘s publications portfolio (section III) and comparisons between the Bank‘s publications and those of other institutions (section IV). In each case, we obtain citation data for the relevant publications. In this section, we explain the construction of our publications databases, and the advantages and disadvantages of our different sources of citation data. We also briefly outline our citation-based bibliometric measures of influence. Publications data Our first database is solely of Bank publications. We constructed this by merging (a) the Bank‘s e-Library3 (which covers most Bank-published books) with (b) the Bank‘s working paper catalogue, (c) books co-published by the Bank and other publishers (mostly university presses whose copyright rules often disallow the inclusion of the publication in the e-Library), and (d) book chapters and journal articles published by commercial publishers and authored (or co- authored) by Bank staff. Our database appears to include virtually all formal publications 2 For details of Google Scholar coverage, see http://scholar.google.com/intl/en/scholar/inclusion.html. Accessed October 31, 2011. 3 The Bank‘s e-Library is to be found at http://e-Library.worldbank.org/. Accessed October 31, 2011. We downloaded the metadata from http://www.worldbank.icebox.ingenta.com/jsp/worldbank/home/librarians in April 2010. As of October 31, 2011, the metadata were at http://elibrary.worldbank.org/librarians. 6 authored by Bank staff since about 1970, as well as some books that have been commissioned by Bank staff though not authored by them. Our database differs from the Bank‘s ‗documents and reports‘ (D&R) database,4 which contains most of (a) and (b), but excludes (c) and (d); D&R also includes a large number of items that are not formal publications, as well as some journal articles that are not actually authored by Bank staff. We began with the Elsevier-owned SCOPUS database.5 This covers a longer time period and a larger set of journals than the Social Science Citation Index (SSCI),6 and includes several non-journal publications, including North-Holland Handbook chapters, 14 of which have been authored or coauthored by Bank staff. We searched for publications in SCOPUS where any of the authors listed in the publication had indicated the World Bank as their institutional affiliation. The SCOPUS database includes articles from 1965. We then combed through the SSCI which dates back only to 1982 and contains mostly journal articles, again looking for articles where World Bank appeared in any of the author‘s address field. Next we searched through EconLit ,7 which dates back to 1969 and contains journal articles but also some book chapters and some working papers; again, we searched for entries where World Bank appeared in the address field of any of the authors. Next we pulled in the Bank‘s entire e-Library, which contains Bank- published books (dating back to 1978) and working papers in the Bank‘s Policy Research Working Paper (PRWP) series (dating back to 1994). Finally, we searched the US Library of Congress catalogue, looking for books published by the Bank but missing from the e-Library; these are typically books jointly published with a university press, presumably omitted from the e-Library for copyright reasons. The Library of Congress books date back to 1974. Finally, we eliminated duplicate titles for each publication type, eliminating punctuation and spaces in the title beforehand to reduce the risk of not identifying duplicates due to different punctuation conventions across the various databases. We did not at this stage eliminate duplicates across 4 The database is available at http://www-wds.worldbank.org/. Accessed October 31, 2011. 5 We collected the metadata from all databases in April 2010. SCOPUS covers over 18,000 titles covering all disciplines including 16,500 peer-reviewed journals, as well as some book chapters, including the North-Holland handbook series. Details of the database are to be found at http://info.scopus.com/scopus-in-detail/facts/. Accessed October 31, 2011. 6 SSCI covers primarily journal articles and is limited to about 2,500 social science journals. Details of coverage are to be found at http://thomsonreuters.com/products_services/science/science_products/a- z/social_sciences_citation_index. Accessed October 31, 2011. 7 Details of EconLit coverage are to be found at http://www.aeaweb.org/econlit/doctypes.php. Accessed October 31, 2011. 7 publication types, so that a title could appear as a working paper and a journal article, and sometimes as a book chapter as well; we show the extent of such duplication below. We use both SCOPUS and the SSCI for our comparisons between the Bank and other institutions. For the most part, these databases comprise journal articles. In each case, we drew records from the online database where the Bank was included in the address field. We also drew records for 21 comparator institutions, namely 14 universities in the UK and US, and seven international agencies working on finance and/or development. In some of our comparisons, we restricted our attention to specific subjects, topics or fields, or specific sets of journals. In our SCOPUS database, all our results relate to the discipline ―economics, econometrics and finance,‖ and we exclude multidisciplinary articles, i.e., ones where ―economics, econometrics and finance‖ is just one of the disciplines. In our SSCI database, we present results for all SSCI subject areas, but also indicate how different institutions vary in their specialization across SSCI subject areas, as well as how the Bank performs relative to other institutions in several SSCI subject areas, including ―economics‖ and ―planning and development.‖ Citation data For the Bank publications database, we obtained citation data from GS for as many records as we could find. For the SSCI comparative database, we report the citation data recorded in the SSCI – these data only reflect influence within SSCI-indexed journals. In the case of SCOPUS, we also report the citation data recorded in the database. However, we also extracted the metadata for each publication and obtained – again, for as many records as we could – citation data from GS. The citation numbers reported by SCOPUS are likely to be lower than the numbers we obtain passing the same records through GS, since the GS citation numbers reflect influence in a larger set of publication types. A comparison of the two allows us to explore whether the Bank is better than other institutions at reaching the broader set of authors of publications not indexed by SCOPUS. As we mentioned above, GS allows us to extend the range of publications for which we obtain citation data, as well as extend the scope of our assessment of influence on development thinking. GS also has the merit of being somewhat more global and timely. GS does have some 8 possibly less attractive features that are worth highlighting. GS does not capture citations in newspaper articles or editorials, magazines, book reviews, web sites, or blogs. To our mind, however, this is not a bad thing, given our objective. Citations in such media are better seen as indicators of visibility, and perhaps even of impact on policymakers and voters. But this need not (and typically probably doesn‘t) coincide with influence on thinking in the relevant profession, which is our focus here. Levitt and Dubner‘s (2005) Freakonomics is a nice example. As of August 2010, this book had been cited only 415 times in GS; but it gets over 6 million hits on a Google search of the internet, including 230,000 hits on blogs alone. The success of Freakonomics is not that it has influenced thinking in the economics profession, but rather that it has popularized some of the new thinking within the profession, especially on incentives, and experimental and quasi-experimental evidence. Many Bank books may well also score well on visibility and even development impact, but less well in influencing development thinking. That does not make them unsuccessful, of course. GS has other features worth noting. One is its scope. SSCI and SCOPUS begin with a defined list of articles and book chapters based on tables of contents from the journals and edited volumes that the citation database indexes. For each of these articles, the database has clean metadata listing authors, affiliations, title, journal, volume, pages, etc. By contrast, it is not clear how GS arrives at its list of publications for which it reports citation data; there is no public list of journals it indexes, for example, although it is known that citation data are provided for all books in the Google Books database. Data quality is another issue. Items for which GS reports citation data sometimes contain inaccuracies. Titles are sometimes incorrect, incomplete, ambiguous, or appear in multiple versions. The messiness of some of the GS data meant we had to do some careful manual checks of citation numbers for some titles, especially book titles, including all of the WDR series and several other Bank book series.8 8 For example, the most cited book published by the Bank – Angus Deaton‘s Analysis of Household Surveys: A Microeconometric Approach to Development Policy – appears three times in a Google Scholar search, once with just the main title, once with the full title, and once with the main title and an altogether different subtitle (presumably from a draft version). The data for the WDR‘s are similarly messy. In such cases, we have aggregated manually citations across items. In general, however, we relied on the automated search tool we developed for the paper, which went through multiple revisions until we felt it was returning sufficiently accurate data. 9 Bibliometric methods We report a variety of bibliometric measures for our various ‗units of analysis‘ – publication types in the case of our intra-institutional comparisons, and institutions in the case of our inter-institutional comparisons. The first is simply the publication count. The rest are based on citations from one or other of our sources, namely SSCI, SCOPUS, and GS. These include the mean and median number of citations per publication, to measure average influence. These measures are independent of scale: a small and large institution could get a high average citation score. Average citations tell us nothing about total influence. The Bank, for example, could have invested all its efforts into producing one stellar publication on one particular topic that has been cited thousands of times. But this hardly constitutes substantial influence. To measure aggregate scholarly influence through citations we use Hirsch‘s (2005) h- index, which aims to capture both the volume of work and its influence. The index is easy to calculate and interpret.9 An h-index of x means that the individual (or institution or journal or book series) has published x items each of which has been cited at least x times. So one cannot get a high h-index simply by publishing lots of papers; they also have to have been influential, as indicated by the citations of other authors. Hirsch argues that this index is a robust and relevant measure of ―...the importance, significance and broad impact of a scientist‘s cumulative research contributions.‖ Naturally the h-index of a researcher tends to rise with years of publishing. Differences between institutions in the age distribution of their staff will affect their h-indices, though this is probably not an important factor in practice. Scale differences clearly matter though. Other things being equal, the more researchers an institution has the higher will be its h- index. So, average citations and the h-index should be looked at alongside one another. 9 The h-index is also rather arbitrary, with no clear theoretical foundation. Ravallion and Wagstaff (2011) develop a more general approach, linking citations to a theoretically consistent measure of influence, and using citation curves to draw inferences about influence. We checked the results below using this more general approach; broadly speaking, the same conclusions emerge as with the h-index, so we do not report the more general results here. Ravallion and Wagstaff (2010) provide detailed results. 10 III. INTRA-INSTITUTIONAL COMPARISONS We start by looking at the Bank‘s publications portfolio, establishing its size and composition in terms of publication types. We also present data on trends in publication volume. In the second part of the section we present data on the differing degrees of influence on development thinking of the different publication types as captured in the citation data. We also highlight the Bank‘s most highly-cited publications. The volume of Bank publications Table 1 shows the breakdown of Bank publications by publication type. Nearly 9,000 journal articles have been published by Bank staff, over four times the number of Bank- published books. We also found nearly 4,000 working papers: most of these are in the Policy Research Working Papers (PRWP) series. Our database includes nearly 5,000 book chapters authored by Bank staff, and 500 edited volumes published by the Bank; of course, not all of the book chapters authored by Bank staff appear in the edited volumes published by the Bank. The bulk of the journal articles come from the SCOPUS database which has broader and longer coverage. The books and edited books, by contrast, come mostly from the e-Library. Book chapters come largely from EconLit, and working papers mostly from the e-Library. Table 1 also shows the extent of duplication. For example, there are 715 journal articles with one duplicate (usually a working paper of the same title), and 22 with two duplicates (usually a working paper and a book chapter). If we delete duplicates, keeping journal articles in preference to books, books in preference to book chapters, etc. (i.e., going down the order of publication types in Table 1), we end up with 8,829 journal articles, 1,912 books, and so on, as indicated in the second-from-last row of Table 1. We are still left with 3,000 working papers that have no duplicate in any other publication type according to the title. This undoubtedly leaves some duplicates since authors may change the title of their paper when submitting or resubmitting to a journal. Eliminating such ―duplicates‖ would need to be done manually, and would involve a lot of guesswork. We have not attempted this task. Figure 1 tracks the composition of the Bank‘s publications since the early 1970s. (To see the trends more clearly, we have presented the data as 7-year moving averages.) We see that the 11 number of books per year has remained fairly constant since the mid-1990s; we see some limited growth in edited volumes and book chapters.10 However, the big growth has come from papers – journal articles and working papers. Some of this growth is artificial, stemming from expanding coverage of the bibliographic datasets; for example, EconLit only started including book chapters in 1987. Nonetheless it is clear that there has been substantial growth since 1990. The final row of Table 1 shows the share of each type of Bank publication that was authored or edited by a (current or past) member of the Bank‘s research department. 11 The figures here refer to the non-duplicated publications, i.e., a working paper and a journal article of the same name are counted just once. Seventy percent of Bank publications since 1995 were produced outside the Bank‘s research department; the department‘s largest shares were in working papers (41%), journal articles (30%), and book chapters (33%). Research staff account for just 10% of the Bank‘s book catalogue, and only a small share of edited volumes (just 2% of the total). These numbers reflect the small size of the research department – it currently employs less than 100 staff, less than 1% of the Bank‘s employees worldwide. But it also brings home the fact that the Bank‘s ―non-research‖ staff members are heavily engaged in producing publications, including journal articles. Citations of Bank publications Table 2 reports citation data from GS for different types of Bank publications. Nearly 70% of Bank publications have been cited at least once. But our database includes over 5,500 publications that have never been cited. Over 40% (2,242) of these are book chapters; another 35% (1,971) are journal articles. Over 83% of book chapters and around 77% of articles were published before 2007 so they ought to have had time to pick up at least one citation. Focusing on the book chapters and aggregating them to their volume sheds only a little light on the causes of non-citation. Over 100 of the un-cited chapters come from just two books, one the collected speeches of the former Bank President James D. Wolfensohn (only 21 of the 103 chapters have 10 Not all book chapters in our database are in Bank-edited volumes. The North-Holland Handbook chapters are an example. 11 Authors are classified as Bank research staff in this paper if they have ever had an appointment in the current research department (known as the Development Research Group (DRG)) or were in its predecessor (the Policy Research Division (PRD)) from 1995 onwards. Long-term and extended-term consultants are included, but short- term consultants are excluded. 12 been cited) and a volume of papers cataloguing ―the transformation of the Bank‖ under Wolfensohn; none of the 19 chapters has ever been cited. Another 95 un-cited chapters come from four collections, one of which has not had a single chapter ever cited.12 The percentage cited and the mean and median citations vary by publication type. The types with the highest percentages ever cited and the highest median citations per publication are journal articles and working papers: only 15% of working papers fail to get a single citation; the figure for journal articles is 22%. Non-citation rates are highest, and median citations lowest, for book chapters and conference volumes; only half of book chapters, for example, have ever been cited. Books lie in the middle, with a median citation of 39 and a percentage with at least one citation of 30%. At least for World Bank publications, then, volume and influence on development thinking appear to go hand-in-hand: the more common types of Bank publication (journal articles and working papers) are also those that appear to have had the greatest influence on thinking among economics and development professionals. This is reflected in the h-indices. World Bank staff have authored 262 journal articles that have been cited at least 262 times, and have authored 158 working papers that have been cited at least 158 times (there is some overlap between these categories). By contrast, they have published only 132 books that have been cited at least 132 times and only 124 book chapters that have been published 124 times. Table 2 also compares the citation statistics of publications authored or edited by research staff with those of publications authored or edited by non-research staff; the comparison is for the period 1995 onwards. In general, the fraction of publications with at least one citation is higher among those authored or edited by research staff; the difference is quite pronounced for books and book chapters, despite books not being a common form of output for research staff. Publications by research staff also score better on median and mean citations: for example, the median citation for a book authored by a research staff member is over four times that of a book authored by a staff member from another department. These differences suggest that there is 12 The volumes in question are: ―An Opportunity for a Different Peru: Prosperous, Equitable, and Governable‖, ―Rural Development, Natural Resources and the Environment: Lessons of Experience in Eastern Europe and Central Asia‖, ―Proceedings of a Conference on Currency Substitution and Currency Boards‖, and ―Global Issues for Global Citizens: An Introduction to Key Development Challenges.‖ None of the 24 c hapters in the rural development volume has ever been cited in the documents covered by GS. 13 more new knowledge in publications from the research department than those from other Bank units. But it also suggests that the publications from units other than the research department are not simply knowledge synthesis – at least some of them are being cited quite heavily. As already noted, journal articles and working papers have been the Bank‘s main channel of influence on thinking among economists and development professionals. The most cited single publication is the paper by Ross Levine and David Renelt, ―A Sensitivity Analysis of Cross-Country Growth Regressions‖, published in the American Economic Review in 1992 which has received 4,040 citations. Among the Bank‘s portfolio of journal articles, one finds some very highly cited papers: Bank staff authored two of the 10 most-cited articles in each of the American Economic Review, the Quarterly Journal of Economics, and the Journal of Human Resources; three of the ten most-cited articles in the Journal of Development Economics, four of the ten most-cited papers in Economic Development and Cultural Change, and the most cited paper in Demography. This seems to go against the idea that the Bank is merely a proselytizer. What are all the Bank‘s highly-cited articles about? In Table 3 we have taken the 200 most-cited Bank journal articles and allocated them to one of the topics listed (three did not fit in any topic). Articles on growth account for 23% of the top-200 articles, and 35% of total citations of the top 200. Finance and the private sector account for similar shares of total citations, but the Bank‘s private sector articles are more cited on average. The other topics appear with less frequency in the top 200, and the articles on them are cited less on average. Nonetheless, the Bank‘s journal article output includes some highly-cited articles in each of the topics listed. The variations in citation rates of Bank articles across the topics may, of course, reflect differences in citation rates of all published articles across these topics; for example, labor articles may generally be cited less than growth articles. We will get a better picture of the Bank‘s successes in section IV where the Bank‘s citation data are presented alongside the citation data for other institutions. Table 4 reports citation data (from GS) for Bank book series containing at least six publications. Top place in terms of mean citations goes to the (non-annual) Policy Research Reports (PRRs) produced by the Bank‘s research department and authored by research staff; these, like the WDR series, are intended primarily for policymakers. The PRR citation counts are highly uneven, as with the other book series. The most cited PRR is Breaking the Conflict Trap 14 (2003) that received 815 citations, while the next most cited – Land Policies for Growth and Poverty Reduction (2003) – received 467 citations. All the rest received less than 200 citations, and half received less than 100. Two have never been cited. The second-most cited book series is Latin America and Caribbean Studies. Books in this series averaged 112 citations per book, with 24 produced to date. Three of these – Beyond the Washington Consensus: Institutions Matter (1998), Beyond the Center: Decentralizing the State (1999), and Determinants of Crime Rates in Latin America and the World: An Empirical Assessment (1998) – have been cited more than the most-cited WDR. The Regional and Sectoral Studies and the World Bank Institute Development Studies series both outrank the WDR in terms of average citations per volume. The WDR ranks only fifth in terms of mean citations. The most-cited one is A Better Investment Climate for Everyone (2005), with 227 citations. The next most-cited was Building Institutions for Markets (2002), which received 155. The 1978 WDR came next with 149 citations, the 1993 WDR Investing in Health came next (114), and then the 2000/2001 WDR Attacking Poverty (95). The bulk of the WDRs have actually been cited rather little; almost half of them (15 out of 32) have received less than 25 citations since their publication, and 10 have received less than 15 citations. One has never been cited at all. We counted 561 Bank publications (318 articles, 96 working papers, 77 book chapters and 64 books) that have higher citations than the most-cited WDR. The 64 books with higher citations include two in the Policy Research Report series, three in the Latin America and Caribbean Studies series, one in the Directions in Development series, one in the World Bank Institute Development Studies series, and one in the Regional and Sectoral Studies series. All appear to be aimed at a similar audience to the WDR, but the citations data suggest they contain more new knowledge than the WDR. Several Bank series have been cited very little, including the World Development Indicators, Global Development Finance (discontinued in 2010), Global Economic Prospects, and the Global Monitoring Report. The World Development Indicators is a compilation of country-level economic and social data and it would be surprising to see it cited other than as a data source. Likewise, the Global Economic Prospects series provides short-term forecasts for global and regional economic growth, and so editions of GEP have a rather short life; the forecasts are often picked up by the mass media, which will not be reflected in GS. Only one of the six editions of the Global Monitoring Report has been cited (with the 2005 report receiving 15 three); again, this might not be surprising given the report‘s objective of surveying recent progress in attaining development goals. The seven editions of the Doing Business series have hardly ever been cited in the documents covered by GS. The most cited single book in the Bank‘s history is not in any of its flagship series, or even by a Bank staff member; it is Angus Deaton‘s (1997) The Analysis of Household Surveys: A Microeconometric Approach to Development Policy, with over 3,000 citations. This book was commissioned by the Bank‘s research department to assist users of household surveys in analyzing the data. This is an example of a class of Bank products that reach out to practitioners – helping to bridge the gap between cutting-edge research methods and routine applications. Its high number of citations suggests that even a publication whose aim is largely to make accessible existing knowledge can end up being highly cited in the types of publication covered by GS. The Deaton volume is an outlier amongst Bank books, as is plain in Figure 4, which gives the number of GS citations received by Bank books plotted against number of publications, ranked by citations. The next most cited volume after Deaton‘s is Patterns of Development 1950- 1970 by Hollis Chenery and Moises Syrquin, published in 1975 and with a total of 831 citations. There are 19 books published by the Bank that received 500 or more citations, while 52 books received more than 300 and 191 received more than 100 (see Table 2). It might be argued that the impact of these knowledge-synthesis publications has to be measured in some way other than citations, even with such a broad publication database as GS. Citations in news articles might be a better indicator to track the influence among policymakers of series like the WDRs, Global Economic Prospects (GEP), the Global Monitoring Reports (GMR), and Doing Business. Some of these certainly receive substantial media coverage. As of June 2010, Doing Business has received 7,330 citations in Google News. Mentions of other Bank reports, however, are much more modest: 1,730 citations of the WDR, 649 citations of GEP, and just 183 citations of GMR; the WDR Google News figure is actually only a little higher than the citations the WDR receives in Google Scholar. Yet some of these reports – including the WDR – do seem to have an influence among policymakers and other development stakeholders. A famous instance in the history of Bank reports is Bill Gates‘ reading an art icle on the 1993 WDR on health. The article showed a chart on mortality in the developing world that shocked Gates. In an interview on the US Public Broadcasting Service, Gates said this was his ―Aha‖ moment—the 16 moment that prompted him to set up the Bill and Melinda Gates Foundation and focus its efforts largely on tackling disease in the developing world. The 1990 WDR on poverty, which outlined a clear strategy for fighting poverty, and the 2004 WDR on service delivery, which proposed a different way of thinking about service delivery, are often claimed to have influenced policymakers and development professionals, including Bank operations staff. IV. INTER-INSTITUTIONAL COMPARISONS The Bank‘s publication volume and citation data are hard to interpret in isolation. In this section, we present some comparative data showing how the Bank fares in comparison to other institutions. We compare the Bank with 14 top universities in the UK and US (Berkeley, Brown, Chicago, Columbia, Cornell, Harvard, LSE, MIT, NYU, Oxford, Princeton, Stanford, UCL and Yale), and seven other development agencies: the African Development Bank (AfDB), the Asian Development Bank (AsDB), the European Bank for Reconstruction and Development (EBRD), the Food and Agricultural Organization (FAO), the Inter-American Development Bank (IADB), the International Monetary Fund (IMF), and the United Nations Development Program (UNDP). The volume of Bank publications compared to other institutions In Table 5 we compare the Bank‘s journal article counts with those of the 21 comparator institutions.13 Columns 1-6 are based on SSCI data; columns 7-8 are based on SCOPUS data. In terms of journal article counts across all fields covered by the SSCI (in practice not just all the social sciences but some medical and science subjects too), the Bank‘s publication count is much smaller than that of all 14 universities. Over the same period (1982 onwards), Harvard published almost 10 times as many journal articles as the Bank. The Bank did, however, publish more than any other development agency; the IMF came closest with nearly 3,000 articles. 13 After creating the data set we realized that we had undercounted EBRD publications because some authors had listed their address as the European Bank rather than EBRD or the European Bank for Reconstruction and Development. 17 The above comparison is rather uninformative, however. The Bank‘s publications are highly concentrated in terms of disciplines and subject areas. If one calculates the Herfindahl index14 with respect to the SSCI subject areas,15 the Bank‘s research is far more concentrated than any of these 14 universities. The Bank‘s Herfindahl index is 0.55, which is very close to the index one would obtain if all of the Bank‘s research output was split equally between two headings (giving an index of 0.5). By contrast, the top US universities have Herfindahl indices in the range 0.06-0.08. Even the London School of Economics (which specializes in economics and other social sciences) has an index of 0.13, much lower than the Bank‘s. The World Bank‘s Herfindahl index is similar to several of the other international agencies, although the IMF‘s index stands out at 0.93. Unsurprisingly, the two areas where the Bank specializes are ―economics‖ (68% of its articles fall into this subject area) and ―planning and development‖ (23%). In economics, the Bank has published more articles covered by SSCI than any university except Berkeley and Harvard, and more than the other development agencies combined. The SCOPUS data in column 7 tell a similar story: focusing on publications in ―economics, econometrics and finance‖ (and excluding multidisciplinary articles), the Bank emerges second to Berkeley in publication counts. In ―planning and development‖ (column 4), the Bank has published more than any other institution in SSCI-indexed journals. The Bank‘s development focus also comes out in columns 5 and 6. Column 5 focuses on articles in the ―top‖ 16 journals publishing on development economics, as identified by Barrett et al. (2000).16 The Bank‘s publication count in these journals is nearly 2.5 times the average among the 14 universities; only Harvard comes close. Barrett et 14 The Herfindahl index is a common measure of concentration given by (in this case) the sum of the squared shares of research output across the N SSCI headings. The index ranges from 1/N (when the shares are equal) to 1 (when they are very highly concentrated such that one heading has all the output). 15 The subject areas used to classify articles in the SSCI appear to include not only the 48 subject areas used by the SSCI (listed at http://science.thomsonreuters.com/mjl/scope/scope_ssci/), but also those used in the Science Citation Index (http://science.thomsonreuters.com/mjl/scope/scope_sci/) and the Arts and Humanities Citation Index (http://science.thomsonreuters.com/mjl/scope/scope_ahci/). All links accessed October 31, 2011. The World Bank‘s articles, for example, fall into 120 subject categories. These are almost completely mutually exhaustive (just three out of over 9,000 records were unclassified), but not mutually exclusive (the percentages across the 120 subject classifications total 180%). In the analysis, we have focused on the top 50 subject areas for each institution; adding more areas led to results that were almost identical. 16 The journals are: World Bank Economic Review, World Development, Journal of Development Economics, Economic Development and Cultural Change, Journal of Development Studies, American Economic Review, Economic Journal, Review of Economics and Statistics, IDS Bulletin-Institute of Development Studies, Journal of Developing Areas, Review of Income and Wealth, Quarterly Journal of Economics, Population and Development Review, Journal of Political Economy, Development and Change and Developing Economies. 18 al.‘s list includes several top non-specialist economics journals in which articles on development make up a small minority such as the Quarterly Journal of Economics. We constructed a longer list of 27 specialized development journals.17 The Bank published 1,702 articles in these journals, ahead of the universities and other international agencies: its publication count in these journals is five times that of the average count among the 14 universities. The development focus of the Bank‘s journal articles comes through in Figures 2 and 3, which use ―word clouds‖ to display the frequency of the 150 most-common words in the titles of articles in the ―economics, econometrics and finance‖ group in the SCOPUS database. The Bank‘s focus on development and developing countries is evident. So too is its emphasis on agriculture, poverty, inequality and rural issues; the words ―agriculture‖, ―poverty‖, ―inequality‖ and ―rural‖ are not on the university word clouds. Health features equally prominently in both word clouds; education and trade feature in both, but are more prominent in the Bank‘s. As Table 6 shows, the Bank is ahead of most of the top universities and all the development agencies in the volume of its papers on these topics, at least in economics. 18 Table 6 also shows the number of papers explicitly mentioning the name of a developing country: Berkeley, Harvard and Oxford all come out ahead of the Bank, but the Bank‘s article count is around 1.5 times the average among the 14 universities and far in excess of the other development agencies. The Bank ranks third, fifth and second respectively in terms of articles explicitly mentioning Africa, China and India in their titles. Harvard outranks the Bank on all three, and on China the Bank‘s count is somewhat less than the average count of the 14 universities. But on Africa and India, the Bank‘s count is respectively 1.6 times and twice the average count among the 14 universities. These figures are consistent with the findings of Das et al. (2009). Using a database of over 76,000 empirical economics papers published since 1985, they found that publications on 17 Our list of 27 specialist development journals is: World Bank Economic Review, World Development, Journal of Development Economics, Economic Development And Cultural Change, Journal of Development Studies, IDS Bulletin-Institute Of Development Studies, Journal of Developing Areas, Population and Development Review, Development and Change, Developing Economies, World Bank Research Observer, World Economy, Journal of Comparative Economics, Journal of African Economies, Economics of Transition, Public Administration and Development, Bulletin of Indonesian Economic Studies, Environment and Development Economics, China Economic Review, Journal of Economic Growth, Development Policy Review, Journal of Asian Studies, International Monetary Fund Staff Papers, African Development Review-Revue Africaine de Developpement, Journal of International Trade & Economic Development and Studies in Comparative International Development. 18 Recall that ‗poverty‘ did not appear in word cloud in Figure 3, which was based on economics, finance and econometrics articles in the SCOPUS database, while the counts in Table 4 for poverty relate to all fields (and are based on SSCI data). 19 specific countries (adjusted for population size) increased with the country‘s GDP per capita with an elasticity of 0.62. (Naturally research on the US is well represented, though not more so than one would expect given the country‘s GDP.) Naturally, the Bank‘s economics research tends to be more skewed toward low and middle-income countries compared to that of the top five US universities (Harvard, MIT, Chicago, Stanford and Princeton). If one focuses on publications on countries except the US in the top 200 economics journals, the Bank has a similar number of publications to these universities. The difference is that half of the Bank‘s publications reported research findings on the poorest 40% of countries (ranked by GDP per capita), while those countries only accounted for about one quarter of the publications in the top five universities (Das et al., 2009). The comparisons between the Bank and universities tell us about institutions‘ total output, not the average output of individual authors. The Bank is, of course, a large institution, with almost 1,000 employees with ―economist‖ in their title. Not all publish in peer-reviewed journals, of course; but as we saw in section IV, many do. By contrast, even in large universities only a fraction of faculty work on economics or development. Our goal in this paper is not to compare the average output of different institutions; it would actually be very hard to do in the case of the Bank, since many non-research staff may publish just a few journal articles in the career – deciding on the denominator would be a Herculean task. However, one comparison that is relatively easy to do, and makes good sense, is to compare universities with the Bank‘s research department. The last column of Table 5 does this: the Bank‘s research department comes in at ninth position. How does the Bank compare in terms of citations? Table 7 takes the same sets of SSCI-indexed articles in Table 5 and reports citations of these articles by articles included in the SSCI database. In Table 8 we report two sets of citation data for the economics articles in the SCOPUS database in Table 5: first we report citations of these articles by SCOPUS-indexed articles; and then we report citations of the same articles by documents included in GS. 20 Comparing Tables 7 and 8 with Table 5 shows that on all economics articles the Bank does worse on the h-index and on average citations per article than on the publication count. On the SSCI data the Bank slips from third position on the publication count to 10th position on the h-index and 11th position on average citations. On the SCOPUS data the Bank slips from second position on the article count to seventh position on the h-index and 10th position on average citations. One explanation of this slippage is that the Bank attaches a greater importance than universities on repackaging existing knowledge – the proselytizing function that Gavin and Rodrik (1995) highlight. If we take the same publications in SCOPUS but switch from citations in SCOPUS-index journals (and books) to citations in documents covered by GS, the Bank‘s position improves from seventh to fifth position on the h-index, and from 10th to seventh position on average citations. The documents covered by GS but not included in SCOPUS may be more willing to cite publications that repackage existing knowledge well. A second piece of evidence consistent with this explanation emerges in the last three columns of Table 8 which, like the last three columns of Table 5, focuses on the period since 1995, and – in the case of the Bank – only includes articles authored by research staff. In Table 5 the Bank‘s research department came ninth on the publication count, while in Table 8 it comes in sixth on the h-index and third on average citations per article. So while the rank of the Bank as a whole worsens when we move from publication volume to citations, the rank of the Bank‘s research department improves. This is consistent with the Bank‘s non-research staff focusing on repackaging existing knowledge, and its research staff focusing on creating new knowledge. There is a second explanation for the Bank‘s slippage down the ‗league table‘ in the move from publication counts to citations, namely that Bank staff are constrained to work on a specific topic, namely development, and on policy-relevant publications within this field. These constraints likely affect citation scores in two ways. First, if research is subject to diminishing returns, in the sense that new discoveries become harder to make the more research effort is applied in a specific field, we can also expect there to be diminishing returns to citations in a given field (i.e.,, the first paper gets the most attention than the second and so on). It follows that, for any given number of publications, a research portfolio that is constrained to be more specialized will tend to have a lower level of total citations (and of course lower average 21 citations per paper) than one that is not. Second, any bias in the profession toward general- interest articles and any bias against development articles specifically will further lower citation rates of Bank staff. The 1980s and ‘90s certainly saw a marked decline in the number of citations to the top field journals relative to the top general journals (Ellison 2002). However, there is some evidence that at least in the case of development economics, research quality may have been rising relative to other fields (Bardhan 2003). Comparisons within the development field and in the areas where the Bank specializes may be fairer. Columns 3 and 4 of Table 7 show citation data for Barrett et al.‘s list of top 16 journals for development. The Bank actually fares worse in terms of average citations than it does on the number of articles in these 16 journals (position 12, as compared to 1). Its position on the h-index is somewhat better (number 7) but still well below its rank on the volume of papers. As mentioned, however, the top 16 journals for development include several general journals, with limited coverage of development. Restricting attention to articles in these 16 journals that also fall within the SSCI field of ‗planning and development‘, the Bank‘s ranking is much higher: second position on average citations, and first position on the h-index. The last two columns of Table 7 present SSCI citation data for our list of the 27 specialized development journals: here the Bank ranks first on both average citations (12) and the h-index (58). The last two rows of Table 6 show the Bank‘s rank on the h-index and average citations per article for specific geographic areas and topics. The picture is mixed. The Bank‘s rank on average citations is typically lower than on the article count, but its rank on the h-index is the same or better than its rank on the article count in five out of the nine cases, namely the three geographic areas, inequality, and health – in all five fields, the Bank finishes second or third on the h-index. Where the Bank‘s position worsens, the drop is mostly one or two positions and the Bank still finishes second or third on the h-index. The exception is trade, where the Bank drops from third on the journal count to 10th on the h-index. V. CONCLUSIONS In addition to its traditional development aid instruments of lending, policy advice and technical assistance, the World Bank increasingly emphasizes it role as a generator and disseminator of new knowledge about development – in short, its role as a ―knowledge bank.‖ 22 The main measureable outputs of this role are the Bank‘s publications. Yet there has been very little rigorous attention given to assessing the Bank‘s published output and its influence on development thinking. This paper has tried to help fill this gap in our knowledge about the Bank‘s effectiveness. Our publication count and citation data tell a mixed story about the Bank‘s publication record. Our publication count data indicate that the Bank is a prolific publisher, not just of books but (increasingly) of journal articles too. The sheer volume of publications is consistent with the claim by Gavin and Rodrik (1995) that ―…the Bank‘s strength lies … in its tremendous powers to spread and popularize ideas that it latches on to.‖ The evidence is more mixed on the Bank‘s contribution to new knowledge and development thinking. The citation data we have assembled (mainly from Google Scholar) suggest the Bank‘s publications portfolio includes a large number of articles and books that have had very little discernable influence. Even the Bank‘s flagship World Development Report appears to fall into this category despite the fact it is sometimes claimed to be innovative. However, in certain niche areas the Bank‘s journal articles do appear to reflect new knowledge. When we focus on development articles, and on economics articles within specific geographic areas such as Africa, China and India, and specific topics such as education, growth, inequality, health, and poverty, the Bank ranks as highly in widely-used citation-based metrics as does in sheer volume of its published output. A number of articles and books have clearly had a considerable impact on development thinking. Indeed, the Bank‘s researchers have authored some of the most cited articles in several of the top economics journals. While this is clearly not typical of the bank‘s publication record, it does suggest that Bank can legitimately claim to be an innovator in development economics, not simply a highly effective proselytizer. 23 Figure 1: Counts of World Bank publications 1973-2006 (7-year moving averages) 500 Journal Articles 450 Books Chapters 400 Conference Proceedings Edited Volumes 350 Working Papers 300 Publications 250 200 150 100 50 0 1970 1975 1980 1985 1990 1995 2000 2005 2010 24 Figure 2: Word cloud for titles of World Bank-authored journal articles Source: Articles in ―Economics, Econometrics and Finance‖ from SCOPUS database. Figure 3: Word cloud for titles of top university-authored journal articles Source: Articles in ―Economics, Econometrics and Finance‖ from SCOPUS database. 25 Figure 4: Citation curve for World Bank books 4000 General The Analysis of Household Surveys: A Microeconometric Approach to Development Policy Policy Research Reports 3500 World Development Reports 3000 Citations in Google Scholar 2500 2000 1500 Breaking the Conflict Trap: Civil War and Development Policy 1000 Land Policies for Growth and Poverty Reduction: Reforming Infrastructure: Privatization, Regulation, and WDR 2005: A Better Investment 500 Climate for Everyone Competition WDR 2002: Building Institutions for Markets WDR 1978 0 0 500 1000 1500 2000 Cumulative number of books, ranked in descending order of citations Table 1: Sources and summary of our bibliographic metadata on World Bank publications Conf. Edited Working Article Book Chapter Total Proceedings Volume paper Data source SCOPUS 5,256 11 14 112 0 106 5,499 SSCI 2,302 0 0 0 0 0 2,302 EconLit 1,276 0 4,655 0 0 73 6,004 e-Library 0 1,717 0 0 506 3,537 5,756 Library of Congress 0 231 0 0 0 0 231 Total 8,834 1,955 4,669 112 506 3,716 19,792 Number of duplicates 0 8,097 1,824 4,392 101 472 3,003 17,902 1 715 117 256 10 33 690 1,821 2 22 1 21 1 1 23 69 Total with duplicates eliminated 8,829 1,900 4,509 102 478 3,003 18,833 Non-duplicate publications since 1995 6,089 1,465 3,235 59 443 2,992 14,295 % by research staff 32 10 33 0 2 41 31 Notes: The dataset was assembled starting with SCOPUS, then SSCI, then EconLit, and so on. The numbers in the table reflect deletion of duplicates within each publication type. Thus from SCOPUS we retrieved 5,268 journal articles authored by one or more Bank staff, from SSCI we retrieved a further 2,296 articles not in the SCOPUS database, etc. The table also shows the number of non-duplicated and duplicated records for each record type. Thus we have 8,097 journal articles without a duplicated in any other publication type. We have 715 journal articles with one duplicate in another record type, typically a working paper (there are 690 working papers with a duplicate in another publication type). The table shows the total for each publication type including duplicates in another publication type. It also shows the totals for each publication eliminating duplicates in other publication types, beginning with journal articles and eliminating duplicates in other publication types, then books, then book chapters, and so on; this process ensures that only working papers not appearing in another publication type are retained. The penultimate row shows the number of these non-duplicate publications published since 1995. The last row shows the share of publications since 1995 that have been authored, co-authored or edited by research staff. 27 Table 2: Citation counts from Google Scholar of World-Bank publications Conf. Article Book Chapter Edited Volume Working paper Proceedings All staff, 1982- N 8,829 1,912 4,509 102 478 3,003 % cited 78% 68% 51% 39% 70% 83% N cites>100 858 175 105 0 18 150 N cites>300 215 47 22 0 2 29 Median citations 10 8 1 0 5 8 Mean citations 42.8 37.9 13.4 4.8 19.2 28.5 h-index 261 133 102 13 50 114 All staff, 1995- N 6,089 1,475 3,235 59 443 2,992 % cited 80% 66% 51% 42% 69% 83% Median citations 11 5 1 0 5 8 Mean citations 45.5 27.5 13.6 7.7 18.5 28.5 h-index 229 95 88 13 46 116 Research staff, 1995- N 1,804 152 1,082 0 10 1,223 % cited 86% 87% 63% 70% 81% Median citations 21.5 25.5 4 4.5 10 Mean citations 76.6 65.1 24.8 20.1 40.8 h-index 181 53 70 5 98 Sources: Authors‘ calculations. Bibliographic data are from the World Bank publications database that we constructed for the paper. Citations for the entries in the database are from Google Scholar. 28 Table 3: Mean and total citation figures for Bank‘s 200 most -cited journal articles, by topic Mean Total Share of Topic N citations citations citations Growth 45 1046 47075 35% Finance 34 485 16494 12% Private sector 16 830 13284 10% Governance 16 621 9940 7% Trade 16 496 7934 6% Education 10 782 7824 6% Inequality 10 642 6420 5% Agriculture 7 699 4894 4% Poverty 11 439 4826 4% Environment 12 389 4670 3% Public sector 8 538 4305 3% Health 6 437 2622 2% Aid 4 593 2373 2% Labor 2 550 1099 1% Total 197 679 133760 Sources: Authors‘ calculations. Bibliographic data are from the World Bank publications database that we constructed for the paper. Citations for the entries in the database are from Google Scholar. Articles were assigned to a (primary) topic manually on the basis of their title and where necessary abstract. 29 Table 4: Comparison of citation counts for different Bank book series Series N % cited Median citations Mean citations Total citations Policy Research Reports 18 89 63.0 140.1 2521 Latin America and Caribbean Studies 27 78 82 100.0 2700 Regional and Sectoral Studies 22 100 64.5 68.1 1498 World Bank Institute Development Studies 42 83 26 47.6 1998 World Development Reports 32 97 28 46.0 1473 Foreign Investment Advisory Service Occasional Papers 14 93 38.5 45.2 633 Trade and Development 28 89 27.5 44.1 1235 Stand alone (no series) 1615 73 9 40.6 65601 Latin American Development Forum 22 77 17.5 34.5 760 Health, Nutrition, and Population 24 79 13.5 26.3 630 Directions in Development 167 78 6 23.2 3879 World Bank Institute Resources 57 82 5 19.5 1109 Law, Justice, and Development 10 100 8.5 14.4 144 Public Sector Governance and Accountability 12 92 10.5 13.2 158 Orientations in Development 13 77 4 9.3 121 Independent Evaluation Group Studies 111 50 0 8.3 921 Doing Business 7 71 4 8.0 56 World Development Indicators 14 7.3 102 New Frontiers of Social Policy 6 67 1.5 4.3 26 Global Economic Prospects 13 38 0 2.7 35 Environment and Development 6 50 0.5 2.0 12 Country Studies 110 29 0 1.7 190 Agriculture and Rural Development 15 27 0 1.3 20 Global Development Finance 17 0.9 15 Annual World Bank Conference on Development Economics 16 12 0 0.6 9 Global Monitoring Report 7 14 0 0.4 3 Berlin Workshop 6 17 0 0.3 2 Sources: Authors‘ calculations from the World Bank publications database we assembled for the paper, and Google Scholar. 30 Table 5: World Bank journal article counts and specialization compared to other institutions SSCI SCOPUS All SSCI 16 top All SSCI Planning and subjects Economics journals for 27 specialist Economics, Econometrics and Institution subjects Development (Herfindahl (N) development development Finance (N) (N) (N) index) economics journals 1995 onwards 1965 onwards and Bank‘s 1982 onwards and all Bank authors and all Bank research group authors only Berkeley 22,627 5.7 3,531 616 704 522 2,798 2,085 Brown 11,091 7.9 863 124 202 174 670 484 Chicago 21,648 6.2 3,175 123 701 276 2,208 1,690 Columbia 29,867 7.4 2,374 259 428 344 1,975 1,486 Cornell 17,731 6.2 2,357 438 360 452 1,847 1,419 Harvard 47,945 7.6 4,861 480 1,118 688 2,347 1,655 LSE 12,075 12.9 3,062 505 591 333 1,744 1,386 MIT 10,505 11.5 2,679 340 745 201 1,585 1,033 NYU 19,619 6.1 2,407 121 320 130 2,158 1,722 Oxford 20,564 5.5 2,649 611 701 647 1,202 1,079 Princeton 9,355 7.4 1,967 113 567 262 1,385 981 Stanford 21,727 6.4 3,027 174 686 353 2,149 1,590 UCL 13,556 8.4 831 193 143 76 910 784 Yale 23,912 8.8 2,280 205 460 316 1,588 1,107 AfDB 47 54.9 17 29 32 AsDB 180 60.1 132 31 56 99 83 EBRD 28 76.7 24 0 8 33 24 FAO 368 27.3 128 41 17 61 34 IADB 146 62.2 108 23 34 158 139 IMF 2,816 92.9 1,942 132 742 123 114 UNDP 119 13.2 15 21 11 19 14 World Bank 5,074 55.0 3,457 1,188 1,343 1,702 2,554 1,125 rank among universities 15 1 3 1 1 1 2 9 rank among donors 1 5 1 1 1 1 1 Sources: Authors‘ calculations from SSCI and SCOPUS data. Data in columns (1), (3) and (4) retrieved August 2010. Data underl ying column (2) and data in column (5) retrieved June/July 2010. 31 Table 6: Journal articles by topic and subject/discipline Institution Africa China India Growth Trade Poverty Inequality Education Health ‗Economics‘ or ‗Development Economics* All Subjects Economics and Planning‘ Berkeley 243 368 191 307 439 227 177 81 93 Brown 142 130 77 143 91 84 86 23 27 Chicago 86 197 112 307 263 180 181 83 103 Columbia 347 712 201 432 840 436 327 84 127 Cornell 191 172 104 253 302 273 209 94 105 Harvard 478 524 259 598 692 446 403 161 298 LSE 245 191 160 310 472 184 221 75 91 MIT 30 119 53 265 267 34 88 50 90 NYU 108 121 48 291 296 121 164 51 33 Oxford 539 217 178 332 337 205 198 85 128 Princeton 106 185 65 151 226 78 126 67 59 Stanford 149 321 82 290 398 93 178 84 126 UCL 188 67 72 103 106 125 179 57 35 Yale 186 169 98 188 267 152 110 63 88 World Bank 352 224 248 747 570 475 273 223 187 Rank among universities Article count 3 5 2 1 3 1 3 1 2 h-index 2 3 2 3 10 3 2 2 2 Av. citations 6 3 2 10 11 9 7 8 9 Note: * includes Economics, Planning & Development, Business, Finance, and Area Studies. Sources: Authors‘ calculations using SSCI data. 32 Table 7: Citation comparisons across institutions using SSCI data 16 development journals 27 specialized development All journals ―Planning and All articles journals Development‖ only Av. Institution Av. Citations h-index Av. Citations h-index Av. Citations h-index h-index Citations Berkeley 18.2 111 33.9 74 12.8 22 7.4 31 Brown 15.3 56 28.4 38 7.7 12 6.8 19 Chicago 33.4 144 54.8 99 3.8 8 4.8 17 Columbia 15.7 86 26.9 55 8.7 10 5.4 22 Cornell 12.0 71 21.0 39 8.5 21 4.9 23 Harvard 26.9 157 46.2 113 9.2 23 8.8 36 LSE 11.0 81 17.3 40 5.3 16 5.6 20 MIT 31.8 136 51.2 102 10.9 17 9.1 24 NYU 17.0 86 33.9 54 2.7 6 5.8 13 Oxford 8.9 79 11.3 41 6.5 24 5.6 29 Princeton 29.1 122 49.3 84 8.3 14 5.9 22 Stanford 23.0 118 41.9 84 8.9 17 8.6 29 UCL 9.4 41 17.2 24 2.5 4 3.3 10 Yale 22.5 99 28.9 56 13.6 19 8.9 28 AfDB 1.1 3 1.0 2 AsDB 4.5 12 7.0 8 EBRD 22.4 11 7.9 4 FAO 2.3 7 2.8 4 IADB 5.0 13 9.4 10 IMF 9.4 52 8.9 36 UNDP 3.0 3 6.5 3 World Bank 13.0 83 18.3 69 13.3 46 12.0 58 rank among universities 11 10 12 7 2 1 1 1 rank among donors 2 1 1 1 Sources: Authors‘ calculations from SSCI data. For lists of top 16 development and top 27 specialized development journals, s ee footnotes 16 and 17. 33 Table 8: Citation comparisons across institutions using SCOPUS / Google Scholar citation data SCOPUS citations Google Scholar citations All years, all Bank authors 1995-, Bank researchers Av. % articles h- Av. % articles h- Av. % articles h- Institution Citations cited index Citations cited index Citations cited index Berkeley 20 81.2 102 74.0 88.1 207 71.8 87.5 179 Brown 13 80.1 46 45.0 89.0 87 52.0 89.7 81 Chicago 27 79.6 105 103.0 88.0 223 84.2 88.5 190 Columbia 16.3 78.7 77 58.2 86.3 161 55.6 85.3 145 Cornell 12.1 77.3 61 40.7 86.0 128 39.1 85.8 112 Harvard 22.9 79.8 111 90.9 88.0 242 94.2 88.8 210 LSE 10.3 73.4 58 43.5 83.9 124 45.2 84.0% 113 MIT 22.9 79.1 85 89.6 87.5 183 80.8 86.7% 144 NYU 16.5 79.2 85 63.7 86.8 178 63.9 86.9% 165 Oxford 10.1 70.7 48 31.4 83.5 95 31.8 83.9% 89 Princeton 18.5 81.0 76 73.8 87.9 165 79.0 89.1% 153 Stanford 20.5 80.7 96 74.5 88.0 200 63.5 87.7% 163 UCL 13.9 75.6 51 46.2 87.7 104 46.7 87.1% 98 Yale 18.6 78.1 74 61.8 86.3 143 49.4 86.2% 114 AfDB 5.9 63.6 11 21.2 81.8 21 15.7 81.8% 4 AsDB 10.2 63.6 8 37.4 80.8 14 20.6 79.5% 21 EBRD 3.0 60.6 7 9.7 81.8 13 33.5 83.3% 12 FAO 7.7 45.9 19 36.8 65.6 41 14.9 79.4% 11 IADB 8.2 67.7 15 51.3 78.5 38 40.5 80.6% 40 IMF 9.5 65.0 6 35.0 84.6 8 54.0 85.1% 37 UNDP 68.4 84.2 34.6 85.7% 6 World Bank 15.8 80.0 82 69.7 88.3 192 82.2 90.2% 154 rank among universities 10 5 7 7 2 5 3 1 6 rank among donors 1 1 1 1 1 1 1 1 1 Sources: Authors‘ calculations from Google Scholar and SCOPUS data. 34 References Bardhan, P. (2003). "Journal publication in economics: A view from the periphery." Economic Journal 113(488): F332-F337. Barrett, C. B., A. Olia and D. Von Bailey (2000). "Subdiscipline-specific journal rankings: whither Applied Economics?" Applied Economics 32(2): 239-252. Chen, S., R. Mu and M. Ravallion (2009). "Are There Lasting Impacts of Aid to Poor Areas?" Journal of Public Economics 93 3-4: 512-28. Das, J., Q.-T. Do, K. Shaines and S. Srinivasan (2009). U.S. and them: the geography of academic research. The World Bank, Policy Research Working Paper Series, 5152. Deininger, K., L. Squire and S. Basu (1998). "Does Economic Analysis Improve the Quality of Foreign Assistance?" World Bank Economic Review 12 3: 385-418. Dollar, D. and V. Levin (2005). "Sowing and reaping: institutional quality and project outcomes in developing countries." World Bank Policy Research Working Paper No. 3524. Dollar, D. and V. Levin (2006). "The Increasing Selectivity of Foreign Aid, 1984-2003." World Development 34 12: 2034-46. Ellison, G. (2002). "The Slowdown of the Economics Publishing Process." Journal of Political Economy 110 5: 947-93. Flores, T. E. and I. Nooruddin (2009). "Financing the peace: Evaluating World Bank post- conflict assistance programs." Review of International Organizations 4(1): 1-27. Gavin, M. and D. Rodrik (1995). "The World Bank in historical perspective." The American Economic Review 85(2): 329-334. Gilbert, C., A. Powell and D. Vines (1999). "Positioning the World Bank." The Economic Journal 109(459): 598-633. Hirsch, J. E. (2005). "An index to quantify an individual's scientific research output." Proceedings of the National Academy of Sciences of the United States of America 102(46): 16569. Knack, S., F. H. Rogers and J.C. Heckelman (2011). "Crossing the threshold: A positive analysis of IBRD graduation policy," Review of International Organizations, in press. Levitt, S. D. and S. J. Dubner (2005). Freakonomics: A Rogue Economist Explores the Hidden Side of Everything. New York, Harper Collins. Marchesi, S. and E. Sirtori (2011). "Is two better than one? The effects of IMF and World Bank interaction on growth." Review of International Organizations 6(3-4): 287-306. Ravallion, M. and A. Wagstaff (2010). The World Bank's publication record. The World Bank, Policy Research Working Paper Series, 5374. Ravallion, M. and A. Wagstaff (2011). "On measuring scholarly influence by citations." Scientometrics 88(1): 321-337. Squire, L. (2000). Why the World Bank should be involved in development research. In: The World Bank: Structure and Policies. Christopher L. Gilbert and D. Vines (ed). Cambridge, Cambridge University Press: 108–31. van de Walle, D. and R. Mu (2007). "Fungibility and the Flypaper Effect of Project Aid: Micro- evidence for Vietnam." Journal of Development Economics 84 2: 667-85. Wagstaff, A. (2011). "Fungibility and the Impact of Development Assistance: Evidence from Vietnam's Health Sector." Journal of Development Economics 94 1: 62-73. 35 Wane, W. (2004). The Quality of Foreign Aid: Country Selectivity or Donors Incentives? The World Bank, Policy Research Working Paper Series, 3325. World Bank (2011). The State of World Bank Knowledge Services: Knowledge for Development 2011. Washington DC, World Bank.