WPS5374 Policy Research Working Paper 5374 The World Bank's Publication Record Martin Ravallion Adam Wagstaff The World Bank Development Research Group Director's office July 2010 Policy Research Working Paper 5374 Abstract The World Bank has produced a huge volume of books had influence on non-academic audiences. Individually- and papers on development--20,000 publications authored journal articles have been the main channel spanning decades, but growing appreciably since 1990. for scholarly influence. The volume of the Bank's This paper finds evidence that many of these publications research output on development is greater than that of have influenced development thinking, as indicated any of the comparator institutions identified, including by the citations found using Google Scholar and in other international agencies and the top universities in bibliographic data bases. However, the authors also find economics. The bibliometric indicators of the quality and that a non-negligible share of the Bank's publications influence of the Bank's portfolio of scholarly publications have received no citations, suggesting that they have had are on a par with, or better than, most of the top little scholarly influence, though they may well have universities. This paper--a product of the Director's office, Development Research Group--is part of a larger effort in the department to assess the impact of World Bank research. Policy Research Working Papers are also posted on the Web at http://econ. worldbank.org. The author may be contacted at mravallion@worldbank.org. The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team The World Bank's Publication Record Martin Ravallion and Adam Wagstaff1 Development Research Group, World Bank 1818 H Street NW, Washington DC, USA 1 We owe an enormous debt of gratitude to Qinghua Zhao; without his programming skills this paper would have been substantially less interesting. We are also grateful to Imran Hafiz for help in retrieving data from online databases. Helpful comments were received from Deon Filmer, Asli Demirguc-Kunt, Dominique van de Walle and Colin Xu. The views expressed here are those of the authors and need not reflect those of the World Bank, its Executive Directors, or the countries they represent. The World Bank (hereafter "the Bank") has been publishing on economic development since the 1950s, though an organized research program only took shape in the 1970s. Since the mid-1990s, the Bank as a whole has been publishing an average of around 100 books per year. The best known of these books--which surely everyone working on development knows--is the Bank's annual World Development Report (WDR). There have been 32 WDRs at the time of writing, going back to 1978. Bank authors have also published some 2,000 other books. But this figure is dwarfed by the Bank's output in scholarly journals--9,000 articles to date. This paper aims to provide an objective bibliometric overview of the Bank's publication record and to compare it to that of other institutions, including the world's top universities. We try to say something about the intellectual influence of all those publications, based on the uses that others have made of them, as revealed by citations. Our assumption is that any publication that has influenced development thinking will get referred to in other publications, and the more influence it has had the more it will get cited. We focus on citations in scholarly publications: scholarly journals, books, conference proceedings, dissertations, research reports. We do not consider citations in newspaper articles or editorials, magazines, book reviews, web sites or blogs. While these can be important indicators of the short-term visibility of a publication, they do not by themselves constitute good evidence of lasting influence on thinking in a field. In short, we are interested in the scholarly influence of the Bank's publication record, not its visibility per se. To track Bank publications, we created our own database, taking records from several online databases, notably the Bank's e-Library, the US Library of Congress catalog, EconLit, Elsevier's SCOPUS database, and Thomson-Reuters Social Science Citation Index (SSCI). In comparing publication counts with other institutions, we focus on SCOPUS and SSCI. We use (in separate exercises) citation data from different sources, including SCOPUS and SSCI, but also Research Papers in Economics (RePEc) and Google Scholar. In the case of Google Scholar, the interface provided by Harzing's Publish or Perish (POP) software is invaluable in constructing bibliometric measures for authors, journals and book series, though it is not suited to the task of assigning citation counts to items on long lists of specific publications, which is where the bulk of our analysis focuses. We had new software developed for this purpose. The data sources (SCOPUS, SSCI, RePEc and Google Scholar) vary greatly in their coverage, which affects which documents have citation data, as well as the citation scores, since 2 the scores are based on citations to a document by other documents in the same database. SSCI covers primarily journal articles and is limited to about 2,500 social science journals. SCOPUS covers over 18,000 titles covering all disciplines including 16,500 peer-reviewed journals, as well as some book chapters, including the North-Holland handbook series. The RePEc database currently covers 280,000 working papers, 900 journals, 2,800 book chapters, 2,400 books and 1,700 software components; however, the citation data are drawn from a subset of these. By contrast, Google Scholar includes journal papers, conference papers, technical reports, pre-prints, post-prints, and abstracts, as well as books and dissertations lodged in the Google Books database.2 This means we are able to get from Google Scholar citation data not just for journal articles (and some book chapters) by Bank authors, but also for books, book chapters and working papers. It also means we are able to get citation data for a broader set of journals including open-access journals, and new and less well-established journals. The comprehensiveness of Google Scholar not only allows us to get citation data for a larger subset of Bank publications, but also to get citations to these publications in a broader subset of other people's publications. Through Google Scholar we are able to probe for influence beyond a narrow set of established journals. Google Scholar also has the merit of being "global" in its reach, including research outputs from everywhere in the world and in multiple languages. Google Scholar is also timelier than SSCI, as it does not wait for formal publication processes to be completed; this is important given how long it typically takes for an economics research paper to appear in print (Ellison, 2002). These differences can matter greatly to citation counts. For example, Adler and Harzing (2009) give the following (possibly rather extreme) example: "The world's most cited computer scientist, Hector Garcia-Molina, has gathered nearly 30,000 citations in Google Scholar, with most of his papers having been published and cited in conference proceedings. In the Science Citation Index, however, Garcia-Molina only receives slightly more than 250 citations, as Thomson Reuters ISI fails to appreciate the importance of timeliness, and chooses not to recognize citations in conference proceedings." Against these advantages of Google Scholar there is one obvious drawback: data quality. The typos and other mistakes in the internet sources complicate the task of tracking down citation numbers and are visible in the (otherwise neat) tabulations and summary statistics produced by POP. For example, according to POP, one of the authors of this paper, Ravallion, 2 Google Scholar searches "journal papers, conference papers, technical reports, or their drafts, dissertations, pre- prints, post-prints, or abstracts" and does not cover "news or magazine articles, book reviews, and editorials". See http://scholar.google.com/intl/en/scholar/inclusion.html. 3 has been publishing for 149 years, since one of his papers is identified in Google Scholar as having been published in 1862; that actually makes him seem young compared to Wagstaff, who has been publishing since 1803 according to POP! This certainly brings down both authors' statistic for citations per year in POP, although other bibliometric measures for authors (discussed later) are robust to such errors. The messiness of some of the Google Scholar data meant we had to do some careful manual checks of citation numbers for some titles. The paper reports on our efforts to tap into these various bibliometric data sources. We make comparisons between the Bank and other institutions, including some of the world's top universities for economics, though we recognize that these institutions have different roles. There are special features of the Bank's work that need to be noted. Naturally that work is focused mainly on the problems of economic development, and the main field is development economics. Most academic economists are unconstrained in their choice of research topics, but that is not the case in institutions such as the World Bank, where the staff members are constrained to work within specific fields, related to the institution's overall objectives. The research topics at the Bank must be relevant to development and Bank researchers are typically specialists working at the intersection of development with another field, such as agricultural economics, inequality, finance, health, education, environmental economics, macroeconomics or international trade. While intellectual freedom is encouraged, the choice of topics is more constrained than is the case for a typical academic. Policy relevance is emphasized, and this does not necessarily lead to papers of interest to academic readers. If we assume that research is subject to diminishing returns, in the sense that new discoveries become harder to make the more research effort is applied in a specific field, we can also expect there to be diminishing returns to citations in a given field (i.e., the first paper gets the most attention than the second and so on). It follows that, for any given number of publications, a research portfolio that is constrained to be more specialized will tend to have a lower level of total citations (and of course lower average citations per paper). This difference between the research output of the World Bank and comparable universities reflects the underlying difference in the roles and objectives of these institutions. There are also arguments suggesting that mainstream academic economics has been somewhat biased against field specialists, including in development economics; Bardhan (2003) provides a carefully argued case for this view. The 1980s and `90s saw a marked decline in the 4 number of citations to the top field journals relative to the top general journals (Ellison, 2002). Yet there is no basis for believing that the relative quality of the top field journals declined; indeed, in the case of development economics, Bardhan (2003) argues that research quality has been rising relative to other fields. Such a bias against field journals would compound the above effects of constraints on diversification, given diminishing returns. These observations speak to the need for an effort in this paper to include comparisons within development field journals, while not neglecting the need to take a broader sweep of the full range of journals, including mainstream economics journals. We will also make comparisons with some non-academic institutions that have similar objectives to the Bank, namely other multilateral development banks. We begin by looking at the volume of Bank publications, after which we turn to the citations those publications have received. The Box summarizes our findings. Finally we try to draw out some implications for assessing the Bank's research output relative to other institutions. Box: Summary assessment of the World Bank's publication record 1. The vast majority of the 20,000 publications by Bank staff or consultants that we have found are not books but journal articles, book chapters and working papers. The Bank is a significant publisher in its own right, but nearly half (45%) of the Bank's publications are published by others, notably in scholarly journals. Yet this sizeable output is not catalogued in the Bank's online Documents and Reports database. 2. There has been substantial growth since 1990 in journal articles and working papers, but no growth in books and only limited growth of book chapters. The Bank has diversified its publications away from books. 3. In the SSCI fields "Economics" and "Planning and Development," the Bank has published more articles than any of the 14 universities except Harvard. Its volume of journal articles in the top development economics journals is nearly 2.5 times the average among the 14 universities; only Harvard comes close. The Bank outranks the 14 universities in terms of the volume of all articles on poverty and on economics articles on development, growth, education, and health. 4. The Bank's journal articles differ noticeably to academic articles in the same journals. Greater focus on development and developing countries is evident, as is the Bank's emphasis on agriculture, poverty, inequality, health, education and rural issues. 5. The emphasis on development has only a small effect on the quality of the journals that Bank authors publish in. The Bank's median article in terms of journal quality is in a journal with an impact factor equivalent to that of the World Bank Economic Review. We find no support for the view that the Bank's authors publish any larger share of their papers in lower quality journals than do academics at most of the top universities in economics. In this respect the Bank is on a par with Columbia and Stanford, and ahead of many others, including the London School of Economics, University College London and Oxford. 6. Around 70% of Bank books are published in a flagship series. The most-cited is the Policy Research Report series. The second-most is Latin America and Caribbean Studies; the third and fourth are the Regional and Sectoral Studies and World Bank Institute Development Studies 5 series. The World Development Reports rank fifth among the Bank book series, with 46 citations per report. However, some of the Bank's flagship publications are largely invisible in Google Scholar, although they are more prominent in Google News. 7. Journal articles by Bank-affiliated authors have clearly had more overall scholarly influence than books, though we do not measure influence on other non-scholarly audiences, including policy makers. A number of journal articles have been cited more than any book volume; for example, over 300 journal articles by Bank authors have been cited more than the most cited WDR. 8. We find that 5,500 Bank publications have never been cited in the vast array of documents covered by Google Scholar. Journal articles and working papers have been cited more in the scholarly literature than books. For example, only 15% of working papers fail to get a single citation; the figure for journal articles is higher at 22%. Nearly 50% of book chapters have never been cited. 9. The Bank compares favorably to the 14 comparator universities in the percentage of cited journal articles: only one university has a lower share of un-cited work. In terms of mean Google Scholar citations per article among articles indexed in the SCOPUS database, the Bank ranks sixth behind Berkeley, Chicago, MIT, Princeton and Stanford. The SCOPUS database contains 192 Bank- authored articles that have been cited at least 192 times--giving an "h-index" of 192; only Berkeley, Chicago, Harvard and Stanford have higher h-indexes. The Bank ranks first among the top 14 universities in the h-index for articles published in development economics journals. 10. Similar results emerge from comparisons with the same institutions using the RePEc data. The Bank ranks first on number of works and journal pages, and also ranks first on downloads and abstract views. On total citations the Bank ranks third. On RePEc's overall index (a harmonic mean of 31 indicators), the Bank ranks 3rd, and ranks first within the development field. 11. The Bank's research department accounts for one third of the Bank's publications; 45% of working papers, 36% of journal articles and 33% of book chapters. Research department staff account for just 11% of the Bank's book catalogue, and hardly ever produce an edited volume. Non-citation rates are lower and citation rates higher among publications authored by research staff. This is true of books and book chapters, as well as the publication types favored by research staff (journal articles and working papers). How much does the Bank publish, where, and on what? Surprisingly perhaps, there is currently no World Bank database that collects even summary data on all the publications of Bank staff. We therefore created as comprehensive a database as we could for the purpose of this paper. We began with the Elsevier-owned SCOPUS database, which covers a larger set of journals than SSCI and also includes various non-journal publications. We searched for publications where the World Bank was listed as the author's affiliation, finding 5,753 publications.3 The bulk of these (5,371) were journal articles, 113 were conference proceedings, and the remaining 134 were a mix of North-Holland Handbook chapters (14), technical and working papers (mostly) from Bank series other than the Bank's Policy 3 Despite efforts to eliminate duplicates at each stage, we discovered some at the end of the construction of the dataset. The figures quoted here are after their deletion. 6 Research Working Paper (PRWP) Series or working paper series hosted at other institutions (107), and books (13).4 We then moved to SSCI adding 2,362 previously undiscovered publications (all journal articles) with the World Bank listed as the author's address. Next we searched EconLit, finding 6,080 further publications authored by World Bank staff. These included 4,658 book chapters, 1,344 journal articles, and 78 working papers not in the PRWP series. We next added items in the World Bank's e-Library not already in our database: this resulted in us adding 3,545 papers in the PRWP series, 506 edited books, and 1,702 books published by the World Bank. Finally, we added from the US Library of Congress catalog 238 books co-published by the Bank and university presses not included in the Bank's e-Library. Our final database contains 20,051 publications, comprising 9,077 journal articles, 4,672 book chapters, 3,730 working papers, 1,953 books, 506 edited volumes, and 113 conference proceedings. There is some double-counting, since a share--we would guess half--of the working papers re-appear as journal articles. Even so, that leaves some 18,000 distinct products. A basic message to emerge from our analysis, therefore, is that the vast majority of the Bank's publications are not books but rather articles, book chapters and working papers. Nearly half (45%) of the publications by Bank authors are actually published by other publishers in scholarly journals. Yet this sizeable output of the Bank's work is not catalogued in its online Documents and Reports database.5 Figure 1 tracks the composition of the Bank's publications since the early 1970s. (To see the trends more clearly, we have presented the data as 7-year moving averages.) We see that the number of books per year has remained fairly constant since the mid-1990s; we see some limited growth in edited volumes and book chapters.6 However, the big growth has come from papers-- journal articles and working papers. Some of this growth is artificial, stemming from expanding coverage of the bibliographic datasets; for example, EconLit only started including book chapters in 1987. Nonetheless it is clear that there has been substantial growth since 1990. How does the Bank's publication count compare with other institutions? In Table 1 we compare the Bank's journal article counts with those of 14 comparator universities--chosen to represent the top universities for economics in the US and UK--and seven comparator 4 Some of the SCOPUS publications were initially classified as "generic", and had to be manually reclassified. 5 The database is available at http://www-wds.worldbank.org/. 6 Not all book chapters in our database are in Bank-edited volumes. The North-Holland Handbook chapters are an example. 7 international agencies working on finance and/or development. Our data are drawn from the SSCI and SCOPUS. In terms of journal article counts across all fields covered by the SSCI, the World Bank is hardly a star performer relative to these 14 universities. Over the same period, Harvard published almost 10 times as many journal articles as the Bank. Indeed, all of 14 universities used in our comparative analysis published more journal articles than the Bank. The Bank did, however, publish more than any other international financial or development institution; the second place went to the IMF, with more than 2,100 articles. However, this comparison is rather uninformative. The Bank's publications are highly concentrated in terms of disciplines and subject areas. If one calculates the Herfindahl index with respect to the SSCI subject areas, the Bank's research is far more concentrated than any of these 14 universities.7 Table 1 makes the comparison. The Bank and the other international agencies are much more specialized than the universities. The Bank's Herfindahl index is 0.55, which is very close to the index one would obtain if all of the Bank's research output was split equally between two headings (giving an index of 0.5). In fact, 91% of the Bank's research output is in the SSCI headings of "economics" (68%) and "planning and development" (23%). By contrast, the top US universities have Herfindahl indices in the range 0.06-0.08. Even the London School of Economics (which specializes in economics and other social sciences) has an index of 0.13, much lower than the Bank's. However, the World Bank's Herfindahl index is similar to the other international agencies; the IMF's index stands out at 0.93. Naturally, the Bank's research is far more concentrated on economics, and development economics in particular. If one focuses on the articles in the SSCI fields "Economics" or "Planning and Development," one finds that the Bank has published more than any university except Harvard; the Bank published over 3,500 articles in these fields, as compared to an average of 2,500 across the 13 comparator Universities; Harvard published 4,400 (Table 1). The SCOPUS data tell a similar story. Focusing on publications in "Economics, Econometrics and Finance" (and excluding multidisciplinary articles), the Bank emerges second behind Berkeley in terms of publication counts. In both the SSCI and SCOPUS data, the Bank's publication count is around 1.5 times the average publication count of these 14 top universities. 7 The Herfindahl index is a common measure of concentration given by (in this case) the sum of the squared shares of research output across the N SSCI headings. The index ranges from 1/N (when the shares are equal) to 1 (when they are very highly concentrated such that one heading has all the output). 8 The development focus of the Bank's journal articles comes through in Figures 2 and 3, which use "word clouds" to display the frequency of the 150 most-common words in the titles of articles in the "economics, econometrics and finance" group in the SCOPUS database. The Bank's focus on development and developing countries is evident. So too is its emphasis on agriculture, poverty, inequality and rural issues; the words "agriculture", "poverty", "inequality" and "rural" are not on the university word clouds. Health features equally prominently in both word clouds; education and trade feature in both, but are more prominent in the Bank's. Table 2 shows the number of papers explicitly mentioning the name of a developing country. Berkeley, Harvard and Oxford all come out ahead of the Bank, but the Bank's article count is around 1.5 times the average among the 14 universities and far in excess of the other development agencies. The Bank ranks third, fifth and second respectively in terms of articles explicitly mentioning Africa, China and India in their titles. Harvard outranks the Bank on all three, and on China the Bank's count is somewhat less than the average count of the 14 universities. But on Africa and India, the Bank's count is respectively 1.6 times and twice the average count among the 14 universities. These figures are consistent with the findings of Das et al. (2009) who, using a database of over 76,000 empirical economics papers published since 1985, found that publications on specific countries (adjusted for population size) increased with the country's GDP per capita with an elasticity of 0.62. (Naturally research on the US is well represented, though not more so than one would expect given the country's GDP.) They found the Bank's economics research to be more skewed toward low and middle-income countries compared to that of the top five US universities (Harvard, MIT, Chicago, Stanford and Princeton). If one focuses on publications on countries except the US in the top 200 economics journals, the Bank has a similar number of publications to these universities. The difference is that half of the Bank's publications reported research findings on the poorest 40% of countries (ranked by GDP per capita), while those countries only accounted for about one quarter of the publications in the top five universities. The Bank's development focus also comes through strongly in the last two columns of Table 2. The penultimate column focuses on articles in the top 16 development economics journals, as identified by Barrett et al. (2000).8 The Bank's publication count in these journals is 8 They include World Bank Economic Review, World Development, Journal of Development Economics, Economic Development and Cultural Change, Journal of Development Studies, American Economic Review, 9 nearly 2.5 times that of the average among the 14 universities; only Harvard comes close. Barrett et al.'s list includes several top non-specialist economics journals in which articles on development make up a small minority. The list also excludes several good development economics journals (including the WBRO). We constructed a longer list of 27 specialized development journals.9 The Bank published 1,702 articles in these journals, ahead of universities and other international agencies: its publication count in these journals is five times that of the average count among the 14 universities. An emphasis on development and specialist development journals might be expected to lead its authors to publish in lower tier journals. Development economics has been a growing field since the 1980s. The extra research output cannot be expected to all make it into the "top journals," which naturally have limited space, and cover all fields of economics. While there may well be some expansion in coverage of growing fields in general economics journals, new field journals can also be expected to emerge, and we have seen this in development. An independent evaluation of Bank research--widely known as the Deaton report, after the chairperson of the panel, Professor Angus Deaton of Princeton University--questioned the value of many of the Bank's research papers, arguing that: "...the problems lie not with the best researchers, and the best journals, but with the long tail of undistinguished work that is directed towards, and appears in, the second tier field journals, or in (some of the) conference volumes." (Banerjee et al., 2006, p.75) However, it is not clear that the Bank's research is different to (say) Princeton in this respect. Research is risky, and any research program can be expected to produce outputs that (for one reason or another) do not make it into the best journals, though may still be worthwhile contributions. The Bank's Chief Economist at the time responded to the above claim by the Economic Journal, Review of Economics and Statistics, IDS Bulletin-Institute of Development Studies, Journal of Developing Areas, Review of Income and Wealth, Quarterly Journal of Economics, Population and Development Review, Journal of Political Economy, Development and Change and Developing Economies. 9 This list includes: World Bank Economic Review, World Development, Journal of Development Economics, Economic Development And Cultural Change, Journal of Development Studies, IDS Bulletin-Institute Of Development Studies, Journal of Developing Areas, Population and Development Review, Development and Change, Developing Economies, World Bank Research Observer, World Economy, Journal of Comparative Economics, Journal of African Economies, Economics of Transition, Public Administration and Development, Bulletin of Indonesian Economic Studies, Environment and Development Economics, China Economic Review, Journal of Economic Growth, Development Policy Review, Journal of Asian Studies, International Monetary Fund Staff Papers, African Development Review-Revue Africaine de Developpement, Journal of International Trade & Economic Development, IMF Staff Papers and Studies in Comparative International Development. 10 Deaton report as follows: "We find no objective evidence that the Bank's research `tail' is longer than that of other leading institutions" (Bourguignon, 2006, p.3). What do the data suggest? In Table 3 SSCI-indexed journals have been weighted by their impact factor (their h-index, computed by the SCImago Research Group based on the SCOPUS database).10 The median quality of the journals in which Bank authors publish is 29 (the h-index of the World Bank Economic Review)--lower than the average of the universities, but higher than the average of the multilateral institutions. However, these are clearly not large differences. If we focus on the 25th percentile--as an indication of the "lower tail"--the quality of the journals that Bank authors publish in is equal to or better than nine of the 14 universities. This does not support the Deaton report's claims about the Bank's "long tail of undistinguished work," at least if this is interpreted as a tendency for the Bank's authors to publish in low-quality journals. It is actually at the other tail where the differences are more pronounced; the 75th percentile of the distribution of the h-indices of the journals in which Bank authors publish in is lower than all but two universities. Bank staff hit top journals less frequently than university faculty, but Bank authors appear in bottom-tier journals less frequently. Table 4 reports publication counts for selected themes on which the Bank has published widely, including poverty, inequality, growth, trade, education and health.11 The Bank is ahead of most of the top universities and all the agencies in the volume of its papers on these topics, except in the fields of education and health if the publications are not restricted to those in economics (hardly surprising given the Bank does not have a medical school!). How much are the Bank's publications cited by others? Clearly, one of the objectives of the Bank in publishing is to influence development thinking. The classic indicator of influence for academic research is the citation count. It is established practice that research and (serious) policy papers cite other papers or books on which 10 An h-index for a journal of 82 (the score for the AER) means that the journal has published 82 articles each of which has been cited at least 82 times (in SCOPUS-indexed journals). The QJE's h-index is 76, the JPE's 62, the Journal of Development Economics 36, the WBER 29, the WBRO 25, EDCC 22, and the Journal of African Economies 13. See http://www.scimagojr.com for details. The h-index data for this table are not the latest data which were released only recently. The broad conclusions are unlikely to be much affected. 11 Recall that `poverty" did not appear in word cloud in Figure 3, which was based on economics, finance and econometrics articles in the SCOPUS database, while the counts in Table 4 for poverty relate to all fields (and are based on SSCI data). 11 they have drawn. Citations are thus considered a reasonably good overall indicator of the influence of academic research and are widely used for this purpose in all fields. Scholarly citations are less relevant for some publications than others. For example, the scholarly community is only part of the intended audience for the Bank's WDRs which also aim to reach policy makers and their advisers globally, as well as journalists and teachers. The degree to which they succeed in influencing these audiences may not be captured well by citations in publications indexed by SSCI and SCOPUS, or even those covered by Google Scholar, which (although more diverse than those indexed by SCOPUS) is still focused on scholarly products. Citations in news articles might be argued to provide a better indicator of the influence of the WDR and other high-profile Bank reports such as Global Economic Prospects (GEP), World Development Indicators (WDI), Global Monitoring Report (GMR), and Doing Business. Some of these certainly receive substantial media coverage. As of June 2010, Doing Business has received a massive 7,330 citations in Google News. Mentions of other Bank reports, however, are much more modest: 1,730 citations of the WDR, 649 citations of GEP, 469 citations of WDI, and just 183 citations of GMR. The WDR Google News figure is actually only a little higher than the citations the WDR receives in Google Scholar. Of course, not all news articles on a Bank report (or any other report) have the same influence. Some--possibly most--get read and then forgotten. But some do get read and acted on. A famous instance in the history of Bank reports is Bill Gates' reading an article on the 1993 WDR on health. The article showed a chart on mortality in the developing world that shocked Gates. In an interview on the US Public Broadcasting Service, Gates said this was his "Aha" moment--the moment that prompted him to set up the Bill and Melinda Gates Foundation and focus its efforts largely on tackling disease in the developing world.12 This is not an isolated case. We have been struck by how much influence some of the WDRs have had in policy circles, such as the 1990 WDR on poverty (which outlined a clear strategy for fighting poverty that has had lasting impact) and the 2004 WDR on service delivery (which has clearly had considerable influence on how people think about service delivery, within and outside the Bank). 12 See http://www.pbs.org/now/transcript/transcript_gates.html. 12 There are clearly channels of influence for such Bank publications that do not involve scholarly citations. Nonetheless, most Bank publications are intended to contribute new and lasting ideas to global debates on development. And in most of the world, academics are not just contributors to this debate but are respected advisors on policy matters; the US is a good example of the revolving-door relationship between academia and policy making. Citations of Bank publications in the work of academics provides hard data on now much the Bank's ideas are getting traction among this group; absence of such citations--even in the face of extensive news coverage--would be rather worrisome. Measuring influence through citations A consensus appears to have emerged in both the physical and social sciences that the h- index proposed by Hirsch (2005) is a useful comprehensive measure of research output, combining both productivity and influence. The index is easy to calculate and interpret. An h- index of x means that the individual (or institution or journal or book series) has published x items each of which has been cited at least x times. So one cannot get a high h-index simply by publishing lots of papers; they also have to have been influential, as indicated by the citations of other researchers. Hirsch (2005) argues that this index is a robust and relevant measure of "..the importance, significance and broad impact of a scientist's cumulative research contributions." As Hirsh (2005) warns, comparisons across disciplines are confounded by differences in their "referencing culture." With reference to the h-index, Zimmermann (2007, p.11) notes that: "This index was developed for physics, where scientists write a lot of papers and also cite rather generously. Some physicists have h above 100, but in economics it is very rare to have an h above 20, mainly due to the fact that economists write fewer, but more involved papers" While our attention here is confined mainly to economics, and development economics in particular, there are differences within the field. Some sub-fields, such as health and finance, tend to get higher citations because of their overlaps with large non-economics fields where citations tend to be higher.13 Naturally the h-index of a researcher tends to rise with years of publishing. Differences between institutions in the age distribution of their staff will affect their h-indices, though this is probably not an important factor. But scale differences clearly matter. Other things being equal, the more researchers an institution has the higher will be its h-index. Given these concerns, we 13 As we will see later, some of the most cited World Bank papers have been on finance and development. 13 will also measure the average number of citations per publication. This has the advantage that it measures quality independently of scale (not penalizing small economics departments for example), though this reflects a disadvantage too: a researcher or institution that published just one well-cited paper could hardly be considered very productive. Thus, to get a complete picture one needs to look at both the h-index and average citations. The idea of the h-index can be generalized to an h(k) index, whereby an index value of x gives the number of publications received k.x citations, where k is some positive parameter (van Eck and Waltman, 2008). It is also useful to define the citation curve, which plots (on the vertical axis) the citations received by the n'th publication ranked in descending order of citations (horizontal axis).14 When the two citation curves do not intersect then the higher curve will have the higher h(k) index over an appropriate range of value of k including k=1.15 We will then say that the higher curve first-order dominates in citations (following Ravallion and Wagstaff, 2010). In such cases, we can make a stronger statement about the influence of the different publication records. We can postulate an "influence function" linking influence to the number of citations. When we see first-order dominance, we can conclude that the publication record with the higher citation curve will have had greater aggregate influence, as assessed by any "influence function". The main conditions for this to hold are that (i) more citations imply greater influence and (ii) the influence function is stable across the publication records being compared (Ravallion and Wagstaff, 2010). The first condition is innocuous. The second assumption is more plausible for comparisons within a given discipline than between disciplines. When citation curves intersect, Ravallion and Wagstaff (2010) show that robust rankings are still possible for a subset of influence functions satisfying a third condition: (iii) the marginal influence from extra citations decreases as the number of citations increases, i.e., the first citation to a given publication has the highest impact, followed by the second, with the extra impact becoming negligible at some very high level of citations. To put the point another way, imagine two publication records, each containing two publications. In A's record, both papers received 50 citations, while in B's, one paper received 100 citations and the other paper received none. Only 14 To make the citation curve continuous one can interpolate between the discrete points, as discussed in Ravallion and Wagstaff (2010). 15 The "appropriate range" is constrained by the lower of the number of publications by the two people (or other units) being compared. More precisely the h(k) index is higher for the higher citation curve for all k>kmin where kmin is the ratio of the number of citations received by the least cited publication by the least published person divided by the total number of publications by that person. 14 one of B's papers is known to have had any influence, while both of A's have demonstrated influence. Under this third condition, A's record is deemed to have higher total influence.16 To test robustness under all three conditions (and some technical conditions given in Ravallion and Wagstaff, 2010), one first graphs (on the vertical axis) the number of publications receiving more than the corresponding number of citations (this time on the horizontal axis) ; this is the inverted citation curve. Next one calculates the area under this curve at each value of citations. If the area under A's inverted citation curve is greater than the area under B's for each citation level, A's publication record has more influence than B's for any stable influence function that is increasing in citations but displays diminishing marginal returns to citations in terms of the influence they generate. In this case, we say that A's publication record second- order dominates B's for this sub-set of influence functions. Citation curve dominance can thus be considered a robust indicator of which publication record has had greater aggregate scholarly influence even when that influence cannot be directly observed. We can construct similar tests for average influence. If one normalizes the standard citation curve by the number of publications (so that the horizontal axis gives percentiles of total publications), then dominance in the normalized curve implies that the higher curve has higher influence per publication for any influence function. Similarly, if the normalized curves intersect, one may still be able to make a robust comparison for the subset of influence functions that display diminishing marginal influence; we simply compare for each level of citations the areas under the normalized version of the inverted citation curve (Ravallion and Wagstaff, 2010). Citations of World Bank-authored publications Our interest is in citations of all Bank publications, not just journal articles for which SSCI and SCOPUS provide citation data. We were also interested in citations in a broad range of documents, including scholarly journals not indexed by SSCI and SCOPUS, and publications other than scholarly journals, such as books, working papers, technical reports, dissertations, and conference proceedings. Most of these are at best partially covered (more often not even covered at all). While citations in such documents may not matter much for university-based authors, they are an important indicator of the degree to which the Bank and other aid agencies achieve the broader influence they strive for. So, in addition to presenting data on citations to Bank 16 Ravallion and Wagstaff (2010) discuss the arguments for and against this third condition. One measure in the literature does not satisfy this condition, though Ravallion and Wagstaff argue against that measure. 15 journal articles in SSCI- and SCOPUS-indexed journals, we also present data on citations in Google Scholar to Bank publications other than journal articles.17 Table 5 reports citation data from Google Scholar for different types of publications. Nearly 30% of Bank publications have never been cited. The "failure rate"--and the mean and median citations per publication--vary by publication type. The types with the lowest failure rates and the highest median citations per publication are journal articles and working papers: only 15% of working papers fail to get a single citation; the figure for journal articles is 22%. Failure rates are highest, and median citations lowest, for book chapters and conference volumes; book chapters, for example, have a failure rate of nearly 50%. Books lie in the middle in terms of failure rates and median citations, with a median citation of 39 and a failure rate of 30%. So we find that, at least for World Bank publications, volume and scholarly influence go hand-in-hand: the more common types of Bank publication (journal articles and working papers) are also more influential in terms of citations. This is reflected in the h-indices. World Bank staff have authored 260 journal articles that have been cited at least 260 times, and have authored 155 working papers that have been cited at least 155 times (there is some overlap between these categories). By contrast, they have published only 135 books that have been cited at least 135 times and only 125 book chapters that have been published 125 times. Figure 4 shows the citation curves for journal articles and books, while Figure 5 shows the normalized citations curve. We see that the journal article citation curve lies everywhere above that of books; if we assume more citations imply more influence (whatever the precise relationship between the two) we can conclude that articles have had greater influence. We can also conclude from the normalized citation curves that articles also have higher average influence (again, assuming only that influence increases with citations). Which Bank books are cited and which are not? Around 70 percent of Bank books are published in a "flagship series," the WDR being the best-known of these series. Others include Directions in Development, and Latin America and Caribbean Studies. Table 6 reports citation data (from Google Scholar) for book series containing at least six publications. We focus first on the Bank's flagship report, the WDR. At the time of writing, our count is that the 32 WDRs, from 1978 to 2009, had a total of 1473 citations in Google Scholar, an 17 We were not actually able to obtain citation data for all 20,000 records in our database. 16 average of 46 per report. The highest citation count is for the 2005 report A Better Investment Climate for Everyone, which received 227 citations. The next highest was for the 2002 report Building Institutions for Markets, which received 155. The 1978 WDR came next with 149 citations, the 1993 WDR Investing in Health came next (114), and then the 2000/2001 report Attacking Poverty (95). The bulk of the WDRs have been cited rather little; almost half the reports (15 out of 32) received less than 25 citations since their publication and 10 of the reports received less than 15 citations. One has never been cited at all. The h-index of the WDRs is 9. In terms of mean citations per publication, the WDR series ranks fifth among the Bank book series. Top place goes to the (non-annual) Policy Research Reports produced by the Bank's research department but targeted to a similar audience as the WDRs. We count 2,338 citations to the 18 PRRs, 130 per report, as compared to the WDR's rate of 46 per report. Like the WDRs the counts are highly uneven. The most cited PRR in Google Scholar is the 2003 report Breaking the Conflict Trap received 815 citations, while the next most cited, Land Policies for Growth and Poverty Reduction (also in 2003), received 467 citations. All the rest received less than 200 citations, and half received less than 100. Two have never been cited. The series has an h-index of 14. The second-most cited series is Latin America and Caribbean Studies. Books in this series averaged 112 citations per book, with 24 produced to date. Three of these--Beyond the Washington Consensus: Institutions Matter (1998), Beyond the Center: Decentralizing the State (1999), and Determinants of Crime Rates in Latin America and the World: An Empirical Assessment (1998)--have been cited more than the most-cited WDR. The third most-cited Bank book series is Regional and Sectoral Studies. The World Bank Institute Development Studies series also outranks the WDR in terms of average citations per volume. Several Bank series have been cited very little, including the World Development Indicators, Global Development Finance (discontinued in 2010), Global Economic Prospects, and the Global Monitoring Report. These reports have broad audiences although it could not be argued that they are not intended to reach, at least in part, the scholarly literature. The World Development Indicators is a compilation of country-level economic and social data, and from purchase and download records it is clear that the volume is widely used. Possibly the citations are there but in a less specific form such as "World Bank data." The rest of the Bank's poorly- cited publications contain essays on important topics which are clearly intended to inform and influence thinking on these issues. The low citation counts for Global Economic Prospects series 17 is less surprising; this series provides short-term forecasts for global and regional economic growth, and so have a rather short life; the forecasts are often picked up by the mass media, which will not be reflected in Google Scholar. Only one of the six editions of the Global Monitoring Report has been cited (with the 2005 report receiving three); again, this might not be surprising given the report's objective of surveying recent progress in attaining development goals. The seven editions of the Doing Business series have hardly ever been cited in the documents covered by Google Scholar, though this series has clearly had a lot of impact on policy makers and civil society, and is more visible in Google News. It is also notable that there are more scholarly background papers to a number of these flagship reports that get cited more in scholarly writings than the flagships themselves. For example, we found that just one of the Background Papers to the 1990 World Development Report has been cited 200 times, more than twice the WDR itself. Arguably the citation counts for the background papers owe much to the publicity provided by the flagship reports. But the more important direct channel for scholarly influence may well be the background papers to these reports, which could (at least in principle) have been written without the report. The most cited single book in the Bank's history is not any of these flagship series; rather it is Angus Deaton's The Analysis of Household Surveys: A Microeconometric Approach to Development Policy (1997), with 2,282 citations. This book was commissioned by the Bank's research department to assist users of household surveys in analyzing the data. This is an example of a class of Bank products that reach out to practitioners--helping to bridge the gap between cutting-edge research methods and routine applications. The book aimed to reach academics and other researchers, and it is reassuring that it clearly proved valuable to them. The Deaton volume is an outlier amongst Bank books, as is plain in Figure 5, which gives the number of Google Scholar citations received by Bank books plotted against number of publications, ranked by citations. The next most cited volume is Patterns of Development 1950- 1970 by Hollis Chenery and Moises Syrquin, published in 1975 and with a total of 831 citations. There are 19 books published by the Bank that received 500 or more citations, while 52 books received more than 300 and 191 received more than 100 (see Table 5). As noted above, journal articles and working papers have been the main channel for the scholarly influence of the Bank's publications. The most cited single publication is the paper by Ross Levine and David Renelt, "A Sensitivity Analysis of Cross-Country Growth Regressions", 18 published in the American Economic Review in 1992 which has received 4,040 citations. The second-most cited paper is the paper by Ross Levine "Financial Development and Economic Growth: Views and Agenda", published in the Journal of Economic Literature in 1997, which has been cited 3,145 times and was brought out as a World Bank working paper in 1996 while Levine was still at the Bank. Another of Levine's papers occupies 3rd place: "Finance and Growth - Schumpeter Might Be Right" (3,145 citations) published in the Quarterly Journal of Economics in 1993. Fourth place goes to the paper "Institutions and Economic Performance: Cross-Country Tests Using Alternative Institutional Measures" by Steve Knack and Phil Keefer, which was published in Economics and Politics in 1995; this has been cited 2,730 times. The fifth most cited paper is another by Knack and Keefer: "Does Social Capital Have an Economic Payoff? A Cross-Country Investigation", published in the Quarterly Journal of Economics (QJE) in 1997, which has received 2,695 citations. The sixth most-cited Bank publication, "Africa's Growth Tragedy" by Bill Easterly and Ross Levine (written while they were Bank Staff), also published in the QJE, received 2,469 citations. In all, 95 papers received 500 or more citations, 220 received 300 or more, and 894 papers received more than 100 citations. While acknowledging that citations are an incomplete metric of impact for the flagship reports, it was still surprising to us how low the WDRs come in the "citation ladder." According to our count, there are 561 Bank publications (318 articles, 96 working papers, 77 book chapters and 64 books) that have higher citations in Google Scholar than the most-cited WDR. The 64 books with higher citations include two in the Policy Research Report series, three in the Latin America and Caribbean Studies series, one in the Directions in Development series, one in the World Bank Institute Development Studies series, and one in the Regional and Sectoral Studies series; all these are clearly aiming to reach a similar audience to the WDRs. At the other extreme, over 5,500 Bank publications have never been cited at all. Over 40% (2,242) of these are book chapters; another 35% (1,971) are journal articles. Over 83% of book chapters and around 77% of articles were published before 2007 so they ought to have had time to pick up at least one citation. Focusing on the book chapters and aggregating them to their volume sheds only a little light on the causes of non-citation. Over 100 of the un-cited chapters come from just two books, one the collected speeches of the former World Bank president James D. Wolfensohn (21 of the 103 chapters have been cited) and a volume of papers cataloguing "the transformation of the Bank" under Wolfensohn; none of the 19 chapters has ever been cited. 19 Another 95 un-cited chapters come from four collections: "An Opportunity for a Different Peru: Prosperous, Equitable, and Governable", "Rural Development, Natural Resources and the Environment: Lessons of Experience in Eastern Europe and Central Asia", "Proceedings of a Conference on Currency Substitution and Currency Boards", and "Global Issues for Global Citizens: An Introduction to Key Development Challenges." None of the 24 chapters in the rural development volume has ever been cited in the documents surveyed by Google Scholar. Comparisons with other institutions Tables 7-9 compare the Bank on citation data to the same 14 universities and seven development agencies in Tables 1-4. In Table 7 we take the same sets of SSCI-indexed articles in Tables 1 and 2, and report citations of these articles by articles included in the SSCI database. In Table 8 we report two sets of citation data for the 27,613 economics articles in the SCOPUS database: first we report citations of these articles by SCOPUS-indexed articles; and then we report citations of the same 27,613 articles by documents included in Google Scholar. We saw in Table 1 that the Bank ranks second among the universities and first among the development agencies in the number of economics articles it publishes. How does the Bank fare in terms of average citations? On the SSCI data (Table 7), the Bank ranks 11th among the universities and second among the development agencies. On the SCOPUS data (Table 8), the Bank ranks 10th among the universities and first among the development agencies. The Bank's position improves when we take the same set of SCOPUS articles and count citations by the larger set of publications included in Google Scholar: the Bank ranks first among the development agencies and seventh among the universities. Table 8 also reports the percentages of articles that received any citations (100 minus the % un-cited). All institutions have an "undistinguished tail" of un-cited articles. These tails are longer for the development agencies than the universities: typically around 35% of articles published by multilaterals are never cited by articles indexed in SCOPUS; the figure for FAO is over 50%. At 20%, the Bank stands apart from the rest of the multilaterals. In fact, the Bank compares favorably with the universities on the SCOPUS citation data, ranking fifth. When we use the Google Scholar citation data, non-citation rates fall for all institutions, but the switch to the broader citation base improves the Bank's relative position. On the Google Scholar data, only 11.7% of the Bank's articles indexed in SCOPUS have never been cited; the Bank has the second-shortest "undistinguished tail" among the universities, ranking just behind Brown. 20 The h-index captures both the volume of publications and their citations. On this, the Bank ranks 11th on the SSCI data (Table 7) and seventh on the SCOPUS data (Table 8). Broadening the publication base to the Google Scholar data takes the Bank to fifth position on the h-index, behind Berkeley, Chicago, Harvard and Stanford. Figure 7 shows the citation curve for the Bank alongside those of Harvard, Oxford, Princeton and Yale;18 panel (b) enlarges the area around the origin to show more clearly the relative positions of the curves. In terms of the h-index, the raking is clear: Harvard has the highest index, followed by the World Bank, then Princeton, Yale and Oxford. However, with sufficiently high value of k, the generalized h-indices start to show re-rankings as there are a number of intersecting citation curves, implying that robust comparisons are impossible if one allows for any increasing influence function. The claim that Harvard has greater aggregate influence than Oxford is robust, as is the claim that the World Bank dominates Oxford in influence. But the rankings of Harvard, Yale and the World Bank are ambiguous; measures of influence that give sufficiently high weight to high citations would put Yale on top, with the World Bank close behind. The ranking of Princeton and Yale also depends crucially on the weights; the higher the weight given to highly cited publications the more Yale is favored. In Ravallion and Wagstaff (2010) we analyze these institutional rankings further by imposing more structure on the function relating citations to scholarly influence. In particular, we find that if we restrict the analysis to functions that exhibit declining marginal influence from citations then a clearer ranking emerges, with Harvard first, followed by Berkeley, World Bank, Princeton, Yale and Oxford. This is exactly the same ranking as the h-index. So, for these comparisons at least, the h-index ranks consistently with the theoretically ideal measured proposed by Ravallion and Wagstaff (2010). Figure 8 gives the normalized citation curves for assessing the robustness of the comparison of average influence per publication. Very few unambiguous rankings are possible for any increasing influence function. The Harvard-Yale ranking switches when high citations are favored, which also favors the Bank over Princeton. In Table 2 we saw that the Bank ranks first in terms of the number of articles it publishes in development. In Table 7 we see that the Bank's articles in the top 16 development economics 18 Ravallion and Wagstaff (2010) provide citation curves and dominance test for a larger set of universities. 21 journals and the top 27 specialized development journals are cited, on average, 13 and 18 times respectively, and the Bank ranks 12th and first respectively by this measure. As previously noted, Barrett et al.'s list includes several general journals, with limited coverage of development. Focusing on articles published in the journals on Barrett et al.'s list but classified by SSCI as "Development and Planning", the Bank ranks second just in average citations. The Bank ranks first on the h-index for the top 16 development economics journals when the focus is on articles in Planning and Development, and first on the h-index for the 27 development journals. The h-index reveals that the Bank has published 83 articles in journals indexed by SSCI each of which has been cited at least 83 times. In terms of this institutional h-index, the Bank ranks ninth among the universities and first among the international agencies. We have noted the Bank's prominence in work on Africa (Table 2). We also found that this work on Africa is heavily cited. The Bank's h-index for Africa research is higher than all universities except Harvard (details available from the authors). In terms of the h-index the Bank ranks third for China research (only behind Harvard and Stanford). And the Bank is only behind Harvard for research on India. We also identify selected themes where the Bank has concentrated its research effort, namely poverty (third rank amongst universities in terms of h), inequality (second), growth (second), trade (ninth), education economics (second), health economics (second); in each of these topics the Bank's h-index is higher than any other international institution working on finance and/or development. Bibliometrics of World Bank authors So far we have focused on World Bank publications; we turn now to their authors. Bank authors compared to authors elsewhere Most journal articles where the World Bank is listed in the author's field or as the author's affiliation are by Bank staff. There are, we think, only a few exceptions, the most visible being the staff of the tropical diseases research program supported jointly by the Bank, WHO, UNICEF and UNDP.19 We have eliminated these (very productive) individuals from our 19 The UNICEF/UNDP/World Bank/WHO Program for Research and Training in Tropical Diseases (TDR) was started in 1975-76 with the two-fold objective of research and development of new and improved tools for the control of major tropical and neglected diseases and the strengthening of national research capabilities in countries where these diseases are endemic. The main diseases covered were: leprosy, leishmaniasis, schistosomiasis, human 22 database. Bank books, by contrast, are not always authored by Bank's staff. Sometimes they are written by outsider experts commissioned specifically to write a volume the Bank feels could be valuable, but wouldn't otherwise get written.20 This prompts an obvious question: Might bibliometric comparisons of the Bank and other institutions be telling us more about the skills of Bank staff in identifying promising topics and consultants than about their skills as authors? In this section we present bibliometric data on Bank authors rather than Bank publications. We compare Bank authors to authors elsewhere using data from the IDEAS project, based at the Department of Economics at the University of Connecticut using the RePEc database.21 In these data, all an author's publications catalogued by RePEc are allocated to her current institution irrespective of where she was when she wrote the piece and when publication occurred. This means, for example, that the Bank "captures" all a staff member's publications from the years before she joined the Bank. It also means that when someone leaves the Bank, all her publications go with her to her new institution. These comparisons give a bibliometric view of the lifetime achievements of authors currently employed at the Bank compared to the lifetime achievements of individuals employed elsewhere. Table 9 reports comparative bibliometric data by institution for authors registered with RePEc. The focus in RePEc is on economics publications. The database includes journal articles but also working papers and some book chapters. The figures in Table 9 report the institution's ranking among the world's top 5% of institutions (242 at the time of writing).22 Among the 242 institutions that rank among RePEc's top 5%, the Bank as a whole ranks first on journal pages, and first on number of works.23 When the page count and the number of works are adjusted by the "quality" of the journal, the Bank slips to ninth and sixth among the 242 institutions. What of citations when calculated on an author basis? The Bank comes third in terms of the total number of citations, but falls to 14th place when the citations are weighted by their "impact factor," which is based on a ranking of the citing journals or volumes in terms of the citations they receive (Table 9). This reflects the fact that the Bank receives relatively more of African trypanosomiasis, Chagas' disease, lymphatic filariasis and onchocerciasis, and malaria. TDR supports over one hundred TDR medical research institutes in IBRD and IDA countries. See http://apps.who.int/tdr/ for further details. 20 Angus Deaton's The Analysis of Household Surveys is an example. 21 A paper by Coupé (2003) has been influential; however, his ranking of institutions was confined to economics departments in universities, although World Bank authors were included. 22 In Table 9 we include all Bank authors affiliated with RePEc irrespective of their department. 23 Adjusting by the number of authors makes no difference. 23 its citations from journals and books that specialize more on development and also tend to be more policy oriented. The RePEc h-index at the institutional level is author-based: an h-index of 20 on RePEc's institutional ranking means that 20 individuals in the institution have an h-index of at least 20, based on citations of the individual's RePEc-processed documents by other documents processed by RePEc. The Bank ranks 18th out of 242 on this metric, with a score of 12 (Table 9). The RePEc/IDEAS database also provides rankings on the number of abstract views and the number of document downloads via RePEc. The Bank ranks first on both indicators. The IDEAS project reports many other variants of the columns in Table 9, and also presents an overall ranking using a harmonic mean of the ranks on 31 indicators. (Citations, for example, are included unadjusted but also adjusted for age, the "quality" of the citing journal, etc.) Table 10 gives the top 20 institutions in all fields of economics; the Bank comes in at number 3, with only the Departments of Economics in Harvard and Chicago ahead of the Bank. Note that the Bank also appears under "Economic Research, World Bank" in ninth place; this is a subset of those authors affiliated to the Bank who identified themselves this way; roughly speaking, these are current and former members of the Bank's research department. Similarly other institutions appear more than once. The IDEAS project has also produced a consolidated ranking, at institutional level rather than departmental level; this allows for the fact that economists are found in more than one place in many universities (including some of the top ones). By this method the World Bank comes in at the seventh slot, with only six universities ahead of the Bank (Harvard, Chicago, Berkeley, Stanford, Columbia and NYU). Again, it should be recalled that the Bank's research is highly concentrated in terms of its fields. If we focus instead on research outputs mapped to the field "development" then the World Bank ranks number one, ahead of all universities. Table 11 gives the rankings corresponding to Table 10 for the field "development" in RePEc. On top of the fact that the Bank's ranking rises, given its specialization, there are a number of other differences with Table 10. Brown University, Cambridge University, Zurich University and the International Food Policy Research Institute join the top 20; the University of California at Berkeley, Columbia, Boston and Toulouse Universities drop out when we focus of the development field in the RePEc database. A number of the top universities in economics (Harvard, MIT, Chicago, Oxford, LSE) are equally prominent in development (and probably other subfields) as they are in economics as a whole. 24 There are other fields of specialization that stand out. In agricultural economics, the Bank ranks second (just behind the Department of Economics at MIT) while in the field of transition economies, the Bank also ranks second, just behind the Department of Economics at the University of Chicago. For research on international trade the Bank again gets the number two slot (just behind MIT). In the field of banking it comes in at third place (behind the IMF and the Federal Reserve Bank of New York). In research on microfinance, it also comes in at third. The regional emphasis of the Bank's work is also evident in the RePEc rankings, broadly consistent with our earlier observations based on Tables 1 and 2. The Bank ranks second for research on South and East Asia, but gets the number one slot for research on China. For research on Central and Western Asia, the Bank ranks third. For Africa the Bank again gets the number two slot (just behind MIT); the same is true for research on international trade. Curiously, the Bank only gets the number 12 ranking for research on "Central and South America." (And here the Bank is only slightly ahead of the Central Bank of Chile.) The ranking variables used by RePEc are likely to be affected positively by the size of the institution. The Bank as a whole benefits in its RePEc ranking by its size; there are 248 authors affiliated to the Bank who are registered in RePEc. By contrast the average number of authors in the top 20 institutions listed in Table 10 is 98. Even so, the Bank is not the largest institution in Table 10, which is the NBER (with 376 registered authors). "Economics Research, World Bank" has 115 authors registered, and five institutions in Table 10 are larger than that. Note, however, that the average citation rate per article is not affected (directly) by the size of the institution. The citations per article received by the Bank's authors in economics puts the Bank in the top 10 amongst the comparator universities, and when we focus on development articles the Bank rises to near the top. RePEc also produces rankings for individual authors. The Bank has a number of staff who figure prominently in these rankings both across all fields and (especially) in the fields of the Bank's concentration. (A number of ex-Bank staff also figure prominently.) The Bank currently has 12 authors with h-indices (based on the RePEc data base) of 12 or higher and all these are under "Economic Research, World Bank." This is typical of the universities and other institutions in the list of the "top 20" in Table 10. There are 19 staff in the Department of Economics at Harvard, that have an h-index of 19 or higher, and there are 25 affiliates of the NBER with h-indices of 25 or higher. But all the rest have indices in the range 9-13. 25 Looking at the 204 individual authors ranked in the top 10% of all authors mapped to the field "development" in RePEc, we find that 27 are current Bank staff, and 48 are current or ex- Bank staff. This too is a somewhat unusual feature of the Bank's publication record: a relatively large share of its research output is produced by a rather small number of staff. A simple way to measure this is to ask: What proportion of the authors registered on the RePEc database has h- indices of 12 or more? For the Bank, the answer is 5%, and it is 11% for Economic Research, World Bank. These proportions are considerably lower than the US universities in Table 4 (though the figure for Economic Research, World Bank, is similar to the non-US universities and the figure for the Bank as a whole is almost the same as for the IMF as a whole). For the US universities in Table 10, the proportion of authors with h-indices over 12 is 24% or higher. The Bank's relatively low density of active publishing researchers amongst authors registered in RePEc (already a small subset of Bank staff), probably reflects the fact that, for the vast majority of the Bank authors, publishing in scientific journals is not an essential part of their jobs, and (outside the Bank's dedicated research department) may not even be rewarded by the Bank's internal assessments of staff performance. Intra-Bank bibliometric comparisons This takes us to the second aspect of authorship we want to explore--the bibliometric variation across authors within the Bank. The Bank has no single unit dedicated to writing Bank publications; staff across the Bank can and do author journal articles, books, book chapters and working papers, typically in addition to their other duties. The Bank's research department is unusual in that its raison d'être is research, although its staff spend a significant part of their time providing advice and support to other Bank units. For the research unit, publications are a key vehicle for disseminating the unit's findings to a broad audience, and the peer-review process is seen as a crucial part of the quality assurance process. The average (mean and median) h-index of the research department's research staff is 18 and the highest is 81 (based on Google Scholar). In this section, we examine the contribution of the research department to the Bank's publication portfolio, and compare the citation data of research staff and other staff. Table 12 breaks down authorship of Bank publications by type into those authored by members of its research department and those authored by other Bank staff (and authors from other institutions). Two-thirds of Bank publications since 1995 were produced outside the Bank's research 26 department;24 the department's largest shares were in working papers (45%), journal articles (36%), and book chapters (33%). Research department staff account for just 11% of the Bank's book catalogue, and hardly ever produce an edited volume (they account for just 2% of the total). There are some variations by the source of publication data. The research staff's share of journal articles culled from EconLit exceeds that from SCOPUS reflecting the emphasis of the research department on economics and the broader coverage of the SCOPUS database. The fact that research staff have authored a higher percentage of books whose details were obtained from the Library of Congress reflects the fact these publications are co-published with university presses and that research staff have more often been involved in these quite scholarly publications than with Bank books generally. The high fraction of research staff authorship of book chapters from the SCOPUS database reflects the fact that most of these are chapters in the prestigious North Holland handbook series (recall that SCOPUS is owned by Elsevier). Finally, the low percentage for research staff on working papers culled from SCOPUS reflects the fact that these papers are often technical papers and discussion papers issued by Bank departments other than the Bank's research complex. What of the differences between Bank research staff and other authors in citation statistics? Table 13 breaks down citation figures for Bank publications between researchers (former and current members of the research department) and non-research staff by each publication type. In general, failure (i.e. non-citation) rates are lower and citation rates are higher among publications by research staff. This is true of books and book chapters, as well as the types of publications (journal articles and working papers) favored by research staff. It is not true, however, of edited volumes, but very few of these are edited by research staff. The gap is much larger for journal articles than for working papers: research staff and non-research staff produce roughly the same number of working papers, and the failure rate is similar, although the median and mean citations are higher for those authored by research staff. Coupled with the fact that more journal articles are published than working papers, this suggests that the higher failure rate of journal articles (as noted in relation to Table 5) is likely to be due to a high failure rate among papers going to journals that were not previously released as a working paper. 24 Authors are classified as Bank research staff in this paper if they have ever had an appointment in the current research department (known as the Development Research Group (DRG)) or were in its predecessor (the Policy Research Division (PRD)) from 1995 onwards. Long-term and extended-term consultants are included, but short- term consultants are excluded. 27 Conclusions Our aim in writing this paper was to document the Bank's publication record and compare it to the records of other institutions, including the world's top universities in economics. We quickly discovered that there is no institutional database that covers the full breadth of Bank publications; the Bank's own Documents and Reports database fails almost completely to capture the largest and fastest growing component of the Bank's publication portfolio--journal articles. These account for almost half of the 20,000 publications we unearthed by merging several databases. Since 1990, the Bank's annual output of articles has grown while annual book and book chapter numbers have stayed fairly flat. Consciously or unconsciously the Bank has been diversifying its publications portfolio away from books. The Bank now ranks second only to Harvard University in terms of the volume of journal articles it publishes each year in economics. Unsurprisingly, the Bank's journal output--like that of other multilateral development agencies--is highly focused on development: the Bank produces two and a half times the output in the development field of the average university in our 14-university comparator group. Despite this focus, the median journal "quality" is only marginally lower among Bank-authored papers than among papers authored by faculty at the 14 universities. The Bank's longish tail of "undistinguished work" (published in lower quality academic journals) is no longer than for most of the top universities. Our interest in writing the paper was not just to examine the volume of publications, but also to assess their scholarly influence, again using data from other institutions to help set the Bank's performance in context. Our use of Google Scholar citation data gives us a chance to probe broadly (getting citation data on books and working papers as well as journal articles) and deeply (picking up citations in new journals as well as established journals, journals from all disciplines, working papers, books, technical reports, dissertations, etc.). In terms of average citations, the Bank's portfolio of journal articles ranks seventh behind Berkeley, Chicago, MIT, Harvard, Princeton and Stanford. This high average influence, coupled with its large size, gives the Bank's portfolio a large aggregate influence: the Bank's h-index of 192 (meaning it has published 192 articles each of which has been cited at least 192 times) is bettered only by Berkeley, Chicago, Harvard and Stanford amongst the 14 universities we consider. Among the Bank's portfolio of journal articles, one finds some very highly cited papers: Bank staff authored 28 two of the 10 most-cited articles in each of the American Economic Review, the Quarterly Journal of Economics, and the Journal of Human Resources; three of the ten most-cited articles in the Journal of Development Economics, four of the ten most-cited papers in Economic Development and Cultural Change, and the most cited paper in Demography. Perhaps not unsurprisingly, the Bank's books--for the most part--get cited less than journal articles. The most-cited book by far is Angus Deaton's The Analysis of Household Surveys: A Microeconometric Approach to Development, a book commissioned by the Bank's research department to guide practitioners in the analysis of household survey data. Among the various Bank book series, the research department's Policy Research Report series fares best in terms of citations, but several other series, including the Latin America and Caribbean Studies and the Regional Sectoral Studies series also fare well. Surprisingly, the World Development Report series fare less well, though they do get some citations. Of course, the intended audience for the WDRs only overlaps partially with the set of people who write in places that Google Scholar reaches. The same might be said of other high-profile series such as Doing Business (which rarely gets cited in scholarly writings but gets over 7,000 hits in Google News) and Global Economic Prospects. The scholarly literature has ignored some 30% of Bank books and almost 50% of book chapters. Some of them may, of course, have had impact without ever getting cited in a document captured by Google Scholar. However, amongst Bank book series that seem to be targeted at similar audiences (going beyond the scholarly community) some of the series get cited quite heavily while some get cited very little and many never get cited at all. One suspects that uneven quality may be an issue here. Perhaps the Bank's gradual shift in its publications portfolio toward peer-reviewed journal articles is no bad thing. 29 Figure 1: Counts of World Bank publications 1973-2006 (7-year moving averages) 500 Journal Articles 450 Books Chapters 400 Conference Proceedings Edited Volumes 350 Working Papers 300 Publications 250 200 150 100 50 0 1970 1975 1980 1985 1990 1995 2000 2005 2010 30 Figure 2: Word cloud for titles of World Bank-authored journal articles Source: Articles in "Economics, Econometrics and Finance" from SCOPUS database. Figure 3: Word cloud for titles of top university-authored journal articles Source: Articles in "Economics, Econometrics and Finance" from SCOPUS database. 31 Figure 4: Citation curves for World Bank books versus journal articles 4000 3000 citations 2000 1000 0 0 2000 4000 6000 8000 publications ranked in descending order of citations Articles Books 32 Figure 5: Normalized citation curves for Bank books versus articles 4000 3000 citations 2000 1000 0 0 .02 .04 .06 .08 .1 cumulative fraction of publications Articles Books 33 Figure 6: Citation curve for World Bank books 34 Figure 7: Citation curves for Google Scholar citations for the World Bank and various universities (a) Full curve 4000 3000 citations 2000 1000 0 0 500 1000 1500 2000 2500 Harvard Oxford Princeton World Bank Yale 35 (b) Blow-up of lower portion 4000 3000 citations 2000 1000 0 0 100 200 300 Harvard Oxford Princeton World Bank Yale 36 Figure 8: Normalized citation curves for World Bank and various universities (a) Full curve 4000 3000 citations 2000 1000 0 0 .2 .4 .6 .8 Harvard Oxford Princeton World Bank Yale 37 (b) Blow-up of lower portion 4000 3000 citations 2000 1000 0 0 .02 .04 .06 .08 .1 Harvard Oxford Princeton World Bank Yale 38 Table 1: World Bank journal article counts compared to other institutions SSCI SCOPUS All SSCI All SSCI Econ and Economics, journals Institution journals Plan/Dev Econometrics (Herfindahl (N) only (N) and Finance index) Berkeley 25,700 5.7 3,435 2,798 Brown 13,135 7.9 816 670 Chicago 20,859 6.2 2,937 2,208 Columbia 29,409 7.4 2,127 1,975 Cornell 18,144 6.2 2,154 1,847 Harvard 49,260 7.6 4,451 2,347 LSE 11,840 12.9 2,953 1,744 MIT 10,442 11.5 2,225 1,585 NYU 19,258 6.1 1,906 2,158 Oxford 31,357 5.5 2,684 1,202 Princeton 12,623 7.4 2,076 1,385 Stanford 22,833 6.4 2,752 2,149 UCL 13,225 8.4 859 910 Yale 24,216 8.8 2,048 1,588 AfDB 43 54.9 18 AsDB 169 60.1 113 99 EBRD 27 76.7 23 33 FAO 363 27.3 102 61 IADB 142 62.2 105 158 IMF 2,132 92.9 1,317 123 UNDP 117 13.2 22 19 World Bank 5,004 55.0 3,553 2,554 rank among universities 15 1 2 2 rank among donors 1 5 1 1 Sources: Authors' calculations from SSCI and SCOPUS data. 39 Table 2: Comparative data on article counts with a specific developing-country focus Articles 16 top mentioning Articles Articles Articles development 27 specialist a developing mentioning mentioning mentioning economics development Institution country Africa China India journals journals Berkeley 1,788 243 368 191 704 522 Brown 688 142 130 77 202 174 Chicago 122 86 197 112 701 276 Columbia 1,555 347 712 201 428 344 Cornell 933 191 172 104 360 452 Harvard 2,824 478 524 259 1,118 688 LSE 1,203 245 191 160 591 333 MIT 446 30 119 53 745 201 NYU 658 108 121 48 320 130 Oxford 2,453 539 217 178 701 647 Princeton 811 106 185 65 567 262 Stanford 1,117 149 321 82 686 353 UCL 812 188 67 72 143 76 Yale 1,206 186 169 98 460 316 Bank 1,757 352 224 248 1,343 1,702 AfDB 22 32 AsDB 69 56 EBRD 6 8 FAO 60 17 IADB 168 34 IMF 266 742 UNDP 52 11 World Bank 1,757 352 224 248 1,343 1,702 rank among universities 4 3 5 2 1 1 rank among donors 1 1 Source: Authors' calculations from SSCI data. 40 Table 3: The "quality" of the journals in which Bank staff and others authors publish Impact factor of journals Institution p25 Median p75 Berkeley 20 34 46 Brown 20 29 46 Chicago 24 41 62 Columbia 20 32 49 Cornell 20 32 36 Harvard 23 36 62 LSE 17 24 44 MIT 22 36 61 NYU 20 34 49 Oxford 17 24 36 Princeton 24 34 61 Stanford 20 34 62 UCL 18 24 41 Yale 22 34 50 AfDB 14 22 46 AsDB 12 19 36 EBRD 23 46 46 FAO 20 20 22 IADB 19 24 36 IMF 18 26 43 UNDP 13 19 24 World Bank 20 29 41 rank among universities 6 11 12 rank among donors 2 2 4 Sources: Authors' calculations from SSCI and SCIMAGO. Notes: Each journal article is assigned the impact factor of the journal in which it is published. Articles are lined up by their journal's impact factor. The Bank's median score of 29 indicates that the median article appeared in a journal of equivalent impact to the WBER (whose impact factor happens to be 29, ahead of the JDE whose impact factor is 26). 41 Table 4: Journal article counts by topic and discipline Development or Developing Growth Trade Poverty Inequality Education Education country and and All and All and All and All and Health and Health and Institution and Economics Economics* Fields Fields Fields fields Economics All Fields Economics Berkeley 229 307 439 227 177 823 81 1,836 93 Brown 74 143 91 84 86 531 23 1,716 27 Chicago 86 307 263 180 181 656 83 1,658 103 Columbia 186 432 840 436 327 2,495 84 6,450 127 Cornell 182 253 302 273 209 661 94 1,365 105 Harvard 277 598 692 446 403 2,064 161 7,564 298 LSE 189 310 472 184 221 255 75 614 91 MIT 134 265 267 34 88 174 50 300 90 NYU 88 291 296 121 164 721 51 1,404 33 Oxford 235 332 337 205 198 852 85 1,437 128 Princeton 74 151 226 78 126 254 67 449 59 Stanford 136 290 398 93 178 1,052 84 2,280 126 UCL 67 103 106 125 179 648 57 2,408 35 Yale 127 188 267 152 110 803 63 3,010 88 Bank 570 747 570 475 273 375 223 485 187 World Bank 570 747 570 475 273 375 223 485 187 World Bank rank 1 1 3 1 3 12 1 13 2 Note: * includes Economics, Planning & Development, Business, Finance, Area Studies. Sources: Authors' calculations. Table 5: Citation counts from Google Scholar of World-Bank publications Working Edited Conference Type Articles Papers Books Volumes Chapters Proceedings N 9,077 3,730 1,953 506 4,672 113 % cited 78 85 70 71 52 42 N un-cited 1,971 576 589 145 2,242 65 N cites>100 894 298 191 24 160 2 N cites>300 220 73 52 4 46 0 N cites>500 95 34 19 3 27 0 Median citations 10 10 8 5 1 0 Mean citations 42.9 38.5 39.0 22.3 20.5 9.7 h-index 262 155 135 56 123 17 Sources: Authors' calculations from Google Scholar Table 6: Comparison of citation counts for different Bank book series Series N % cited Median citations Mean citations Total citations Policy Research Reports 18 89 59.5 129.9 2338 Latin America and Caribbean Studies 27 78 82 100.0 2700 Regional and Sectoral Studies 22 100 64.5 68.1 1498 World Bank Institute Development Studies 42 83 26 47.6 1998 World Development Reports 32 97 28 46.0 1473 Foreign Investment Advisory Service Occasional Papers 14 93 38.5 45.2 633 Trade and Development 28 89 27.5 44.1 1235 Stand alone (no series) 1,624 73 9 41.3 66991 Latin American Development Forum 22 77 17.5 34.5 760 Health, Nutrition, and Population 24 79 13.5 26.3 630 Directions in Development 167 78 6 23.2 3879 World Bank Institute Resources 57 82 5 19.5 1109 Law, Justice, and Development 10 100 8.5 14.4 144 Public Sector Governance and Accountability 12 92 10.5 13.2 158 Orientations in Development 13 77 4 9.3 121 Independent Evaluation Group Studies 111 50 0 8.3 921 Doing Business 7 71 4 8.0 56 World Development Indicators 14 7.3 102 New Frontiers of Social Policy 6 67 1.5 4.3 26 Global Economic Prospects 13 38 0 2.7 35 Environment and Development 6 50 0.5 2.0 12 Country Studies 110 29 0 1.7 190 Agriculture and Rural Development 15 27 0 1.3 20 Global Development Finance 17 0.9 15 Annual World Bank Conference on Development Economics 16 12 0 0.6 9 Global Monitoring Report 7 14 0 0.4 3 Berlin Workshop 6 17 0 0.3 2 Sources: Authors' calculations from Google Scholar. Table 7: Citation comparisons across institutions using SSCI data Top 16 development journals "Planning and Top 27 development All journals All articles Development" journals only Av. h- Av. h- Av. h- Av. h- Institution Citations index Citations index Citations index Citations index Berkeley 18.2 111 33.9 74 12.8 22 7.4 31 Brown 15.3 56 28.4 38 7.7 12 6.8 19 Chicago 33.4 144 54.8 99 3.8 8 4.8 17 Columbia 15.7 86 26.9 55 8.7 10 5.4 22 Cornell 12.0 71 21.0 39 8.5 21 4.9 23 Harvard 26.9 157 46.2 113 9.2 23 8.8 36 LSE 11.0 81 17.3 40 5.3 16 5.6 20 MIT 31.8 136 51.2 102 10.9 17 9.1 24 NYU 17.0 86 33.9 54 2.7 6 5.8 13 Oxford 8.9 79 11.3 41 6.5 24 5.6 29 Princeton 29.1 122 49.3 84 8.3 14 5.9 22 Stanford 23.0 118 41.9 84 8.9 17 8.6 29 UCL 9.4 41 17.2 24 2.5 4 3.3 10 Yale 22.5 99 28.9 56 13.6 19 8.9 28 World Bank 13.0 83 18.3 69 13.3 46 12.0 58 AfDB 1.1 3 1.0 2 AsDB 4.5 12 7.0 8 EBRD 22.4 11 7.9 4 FAO 2.3 7 2.8 4 IADB 5.0 13 9.4 10 IMF 9.4 52 8.9 36 UNDP 3.0 3 6.5 3 World Bank 13.0 83 18.3 69 13.3 46 12.0 58 rank among universities 11 10 12 7 2 1 1 1 rank among donors 2 1 1 1 Sources: Authors' calculations fron SSCI data. Table 8: Citation comparisons across institutions using SCOPUS and Google Scholar data Scopus articles and Scopus Scopus articles and Google citations Scholar citations Av. % articles Av. % articles Institution h-index h-index Citations cited Citations cited Berkeley 20 81.2 102 74.0 88.1 207 Brown 13 80.1 46 45.0 89.0 87 Chicago 27 79.6 105 103.0 88.0 223 Columbia 16.3 78.7 77 58.2 86.3 161 Cornell 12.1 77.3 61 40.7 86.0 128 Harvard 22.9 79.8 111 90.9 88.0 242 LSE 10.3 73.4 58 43.5 83.9 124 MIT 22.9 79.1 85 89.6 87.5 183 NYU 16.5 79.2 85 63.7 86.8 178 Oxford 10.1 70.7 48 31.4 83.5 95 Princeton 18.5 81.0 76 73.8 87.9 165 Stanford 20.5 80.7 96 74.5 88.0 200 UCL 13.9 75.6 51 46.2 87.7 104 Yale 18.6 78.1 74 61.8 86.3 143 AfDB 5.9 63.6 11 21.2 81.8 21 AsDB 10.2 63.6 8 37.4 80.8 14 EBRD 3.0 60.6 7 9.7 81.8 13 FAO 7.7 45.9 19 36.8 65.6 41 IADB 8.2 67.7 15 51.3 78.5 38 IMF 9.5 65.0 6 35.0 84.6 8 UNDP 68.4 84.2 World Bank 15.8 80.0 82 69.7 88.3 192 rank among universities 10 5 7 7 2 5 rank among donors 1 8 1 1 8 1 Sources: Authors' calculations from Google Scholar and SCOPUS data. 46 Table 9: Ranks amongst 242 institutions based on RePEc data Journal No. works pages Journal No. adjusted Total Abstract Down- Institution adjusted for h-index pages works for impact citations views loads impact factor factor Berkeley 7 9 7 3 6 6 6 7 Brown 44 67 26 34 31 48 44 38 Chicago 10 28 4 9 2 14 7 4 Columbia 46 65 28 29 22 33 38 34 Cornell 125 154 73 77 94 44 128 135 Harvard 4 5 1 1 1 2 2 2 LSE 73 69 34 35 35 33 63 55 MIT 13 15 3 4 5 10 8 8 NYU 11 18 8 8 7 12 10 9 Oxford 2 4 6 7 10 15 4 5 Princeton 9 11 5 5 4 5 9 10 Stanford 36 47 13 19 16 27 34 31 UCL 26 25 23 24 33 23 30 34 Yale 79 72 41 42 54 22 65 63 IADB 259 121 300 194 259 121 89 91 IMF 5 3 19 11 5 3 3 3 World Bank 1 1 9 6 3 18 1 1 Sources: RePEc. Numbers are ranks among 242 institutions. Data not available for other finance/development institutions besides IADB and IMF. RePEc defines h-index of x as saying that x authors affiliated to that institution had an h-index of x or higher (Zimmerman, 2007). 47 Table 10: Top 20 institutions using the 31 RePEc criteria based on research output and citations in all fields of economics Rank 1 Department of Economics, Harvard University (53) 2 Department of Economics, University of Chicago (42) 3 World Bank Group (248) 4 Department of Economics, Massachusetts Institute of Technology (MIT) (26) 5 Department of Economics, Princeton University (43) 6 Department of Economics, University of California, Berkeley (40) 7 London School of Economics (LSE) (174) 8 Department of Economics, Oxford University (137) 9 Economics Research, World Bank (115) 10 Department of Economics, New York University (46) 11 Booth School of Business, University of Chicago (43) 12 International Monetary Fund (IMF) (215) 12 National Bureau of Economic Research (NBER) (376) 14 Department of Economics, Columbia University (50) 15 Department of Economics, Stanford University (44) 16 Kennedy School of Government, Harvard University (48) 17 Department of Economics, Boston University (54) 18 Wharton School of Business, University of Pennsylvania (56) 19 Toulouse School of Economics (TSE) (107) 20 Graduate School of Business, Columbia University (40) Source: RePEc website: http://ideas.repec.org/top/. Numbers in parentheses are the numbers of registered authors in RePEc. 48 Table 11: Top 20 institutions using over the 31 RePEc criteria based on research output and citations in the field of development economics Rank 1 World Bank Group, Washington 2 Department of Economics, University of Chicago, Chicago 3 Economics Research, World Bank Group, Washington 4 Department of Economics, Massachusetts Institute of Technology 5 Department of Economics, Harvard University 6 Department of Economics, Brown University 7 Kennedy School of Government, Harvard University 8 Department of Economics, Oxford University 9 London School of Economics (LSE), London 10 Department of Economics, New York University 11 Wirtschaftswissenschaftliche Fakutät, Universität Zürich 12 Department of Economics Department, London School of Economics 12 International Food Policy Research Institute (IFPRI) 14 International Monetary Fund (IMF), Washington Institut für Empirische Wirtschaftsforschung (IEW), 15 Wirtschaftswissenschaftliche Fakutät, Universität Zürich, Zürich 16 Graduate School of Business, Stanford University Center for International Development, Kennedy School of Government, 17 Harvard University, Cambridge 18 National Bureau of Economic Research (NBER) 19 Faculty of Economics, University of Cambridge 20 Woodrow Wilson School, Princeton University Source: RePEc website: http://ideas.repec.org/top/ 49 Table 12: Research department's share of Bank publications since 1995 by data source Journal Conference Edited Working Source Book Chapter Total Article Proceedings Volume Papers Congress 33% 33% EconLit 41% 33% 47% 35% SSCI 37% 37% Scopus 29% 14% 78% 0% 23% 28% eLibrary 10% 2% 46% 33% Total 32% 11% 33% 0% 2% 45% 32% Source: Authors' calculations from each data source. 50 Table 13: Citation counts of Bank publications by type and author's department Non-research staff Research staff All WB staff Article N 6,603 2,474 9,077 % cited 75 86 78 median citations 8 22 10 mean citations 30.5 75.8 42.9 Book N 1,759 194 1,953 % cited 68 88 70 median citations 7 28 8 mean citations 34.5 79.1 39.0 Chapter in book N 3,374 1,298 4,672 % cited 47 64 52 median citations 0 4 1 mean citations 11.9 43.0 20.5 Conference proceedings N 112 1 113 % cited 43 0 42 median citations 0 0 0 mean citations 9.8 0.0 9.7 Edited volume N 496 10 506 % cited 71 70 71 median citations 5 5 5 mean citations 22.3 20.1 22.3 Working paper N 2,042 1,688 3,730 % cited 85 84 85 median citations 9 15 10 mean citations 25.5 54.2 38.5 Total N 14,386 5,665 20,051 % cited 69 80 72 median citations 5 14 7 mean citations 25.5 61.8 35.8 Sources: Authors' calculations. 51 References Adler, Nancy and Anne-Wil Harzing. 2009. "When Knowledge Wins: Transcending the Sense and Nonsense of Academic Rankings," Academy of Management Learning and Education 8(1): 72-95. Banerjee, Abhijit, Angus Deaton, Nora Lustig and Ken Rogoff. 2006. An Evaluation of World Bank Research, 1998-2005. World Bank. Bardhan, Pranab. 2003. "Journal Publication in Economics: A View from the Periphery," Economic Journal 113: F332-F337. Barrett, Christopher B., A. Olia and D. Von Bailey. 2000. "Subdiscipline-Specific Journal Rankings: Whither Applied Economics?" Applied Economics 32(2): 239-52. Bourguigonon, Francois. 2006. "Chief Economist's Response: An Evaluation of World Bank Research, 1998--2005," World Bank. Coupé, Tom. 2003. "Revealed Performances: Worldwide Rankings of Economists and Economics Departments," Journal of the European Economic Association 1(6): 1346- 1366. Das, Jishnu, Quy-Toan Do, Karen Shaines and Sowmya Srinivasan. 2009. "U.S. and Them: The Geography of Academic Research." Policy Research Working Paper 5152, World Bank, Washington, DC. Ellison, Glenn. 2002. "The Slowdown of the Economics Publishing Process," Journal of Political Economy, 110(5): 947-93. Hirsch, Jorge E. 2005. "And Index to Quantify an Individual's Scientific Research Output." Proceedings of the National Academy of Sciences 102(46): 16569-16572. Ravallion, Martin and Adam Wagstaff. 2010. "On Measuring Scientific Influence," Policy Research Working Paper, World Bank. van Eck, Nees Jan and Ludo Waltman. 2008. "Generalizing the h- and g-indices," Journal of Informetrics 2: 263-271. Zimmermann, Christian. 2007. "Academic Rankings with RePEc," Working Paper, Department of Economics, University of Connecticut. 52