 |
 |
 |
|
| home | CME | help | feedback | subscriptions | archive |
|
© 2003 The Journal of Bone and Joint Surgery, Inc.
Understanding the Limitations of the Journal Impact Factor1 Orthopaedic Research Unit, Repatriation General Hospital, Daws Road, Daw Park, South Australia 5041, Australia. E-mail address: andrew.kurmis{at}flinders.edu.au
Investigation performed at the Orthopaedic Research Unit, Repatriation
General Hospital, South Australia, Australia
The impact factor, a simple mathematical formula reflecting the number of citations of a journal's material divided by the number of citable materials published by that same journal, has evolved to become one of the most influential tools in modern research and academia. The impact factor can be influenced and biased (intentionally or otherwise) by many factors. Extension of the impact factor to the assessment of journal quality or individual authors is inappropriate. Extension of the impact factor to cross-discipline journal comparison is also inappropriate. Those who choose to use the impact factor as a comparative tool should be aware of the nature and premise of its derivation and also of its inherent flaws and practical limitations.
The impact factor was first described in 1955 by Dr. Eugene Garfield1-4 and was used in the early 1960s to help select journals for what would evolve to become the Science Citation Index1. The Science Citation Index, a commercial property of the Institute of Scientific Information (Philadelphia, Pennsylvania)5-7, is used to generate the Journal Citation Reports, produced annually. The Journal Citation Reports list the total number of citations of materials published in a select index of journals during the preceding year. The Institute of Scientific Information also produces a series of other commercial products of value to researchers and academics, including several other citation indexes (e.g., Social Sciences Citation Index, Arts and Humanities Citation Index, and Web of Science), current awareness products (e.g., Current Concepts Connect), and popular information management tools (e.g., EndNote and Reference Manager). Since the first publication of the Journal Citation Reports in 19728, individuals and institutions have been trying to rank journals with use of the data provided in those Reports. However, direct comparison between journals on the basis of the total number of citations alone is an inappropriate measure influenced by a number of factors, such as journal format and content, appropriateness of article classification, and discipline-specific citation tendencies. Thus, a relative adjusted score was sought to allow direct quantitative comparison. Garfield's previously described impact factor was loosely adopted to suit this purpose in the late 1970s, but it did not come into true vogue until two decades later (the late 1990s)3. The original intention for the use of the impact factor was to allow comparison between the citation rates of journals1,6,9. This has proven invaluable for researchers and librarians in the selection and management of journals7,10. The application of this tool evolved into a means with which to assess the quality of the journals themselves9,11, on the basis of the premise that a higher rate of citation indicated higher journal quality. Furthermore, the misuse of this calculation has, in recent years, widened to include evaluation of the quality of individual papers and even individual authors1,6,9,12,13. There is evidence to suggest that some academic assessment committees and institutional promotion, tenure, and funding bodies have also adopted this tool for evaluation of individual researchers or research groups3,5,8,10,11,14,15, although just how widespread this practice is remains unclear. A fundamental understanding of how the impact factor is derived highlights how inappropriate these applications are. Those who use it in such a manner clearly fail to either understand its true derivation or appreciate the original intended use1. The purpose of this paper is to introduce the impact factor to those not intimately familiar with it and to discuss some of the biases and limitations of its application.
The impact factor reflects the number of citations of a journal's material in the preceding two-year period divided by the number of citable materials (source items) published by that same journal within the same period1,2,5,7,13,16-20. The arbitrary selection of a two-year reference period has been the subject of much debate3,7,17, but, while other time-periods have been reported, the two-year period remains the standard frame of reference. While this inclusion period may serve the purpose of reflecting the immediacy of a journal's impact, the scientific value of this time-constrained index is arguably suboptimal, with many important papers achieving their maximal scientific impact outside of this time-frame. Several key factors influence the immediacy of scientific impact1,17,18, an issue that is discussed later in this article. The appropriateness and exact nature of the means with which citations and citable materials are defined for the calculation of the impact factor has also been heavily debated7. For the purposes of calculating the impact factor, a recorded citation refers to a formal reference published in a journal included in the Science Citation Index database. However, distinguishing so-called source from non-source items is a little more ambiguous. The Journal Citation Reports define source items as being original research papers, technical notes, and reviews and papers presented as proceedings, and these solely make up the denominator of the impact factor calculation2,21 (Fig. 1). Non-source items, including letters, news stories, abstracts, book reviews, and editorials, are not included in the denominator of the impact factor equation but may be included in the numerator1,9,18,22,23. For example, if Journal X published one technical note (a source item) and one book review (a non-source item) over the previous two years, and each of these items was then cited only once in journals included in the Science Citation Index, the impact factor for Journal X would be 2.0 (two citations/one source item), even though the source item itself (and, in fact, each item published) had been cited only once. Introduction of a second book review, also cited once, causes the impact factor to jump to 3.0 (and so on). Thus, a potential area of manipulation of the impact factor is introduced.
Unscrupulous editors and authors wishing to raise the impact factor of their journal can take advantage of several such loopholes in the calculation method9,23,24. By increasing the number of non-source materials published in each edition of the journal, the editor can increase the potential pool for citation (the numerator) without increasing the denominator. A similar effect can be achieved by increasing the ratio of non-source to source items (although the total number of items may remain the same). For example, encouraging letters to the editor containing direct citations of the articles being discussed is an easy way to increase the number of citations without increasing the number of source items within the journal. Selecting articles for publication on the basis of the likelihood of a higher citation rate can also dramatically influence the impact factor. For example, review articles3,8,9,18,23,25, minireviews9, and technical reports5 are all recognized as having a higher average rate of citation than original research papers22. Thus, increasing the number of review articles and technical reports per year and limiting the number of original research papers can bolster the impact factor. The strong influence of these articles is clear when it is considered that journals that have been historically assigned the highest impact factors in each field tend to be the review journals. These journals may have a high impact factor even though they may publish no new scientific material at all. Other well-recognized ways of increasing the impact factor of a journal include increasing the rate of self-citation, whereby articles published in a given journal make formal reference to other articles previously published in the same journal. Another method is the encouragement of "salami" publishing3,26, whereby research data are divided into "ridiculously thin slices"3 or manuscripts are broken down into "least publishable units."19 Both of these strategies allow the generation of more publishable work. While some editors and journals have publicly opted not to participate in this impact factor game5,24,27 and have openly criticized others for doing so, there is evidence suggesting that many have already succumbed to the international pressure of the impact factor. The most blatant of this evidence is editorial feedback or explicit requests to authors submitting papers that they include journal self-citations5,9,11,12,22,24,27. There are of course many other, more subtle methods for increasing the impact factor; however, as this paper is not intended to be a recipe for the unscrupulous, they will not be discussed here. The appropriateness and consistency of categorization of materials (as source or non-source items) have long been considered ambiguous. Manual or automated classification protocols are not infallible2, in large part because of the sheer volume of citations recorded and the different styles with which individual journals define and present similar content material1. For example, the style, format, and content of a letter to the editor in one journal may be very different from that in another. Minor errors in the preparation of article reference list information (basic typographical mistakes, inconsistent spelling of surnames and use of initials, and so on)6,8,18 may also inadvertently bias citation identification and recording. For example, a citation listing the first author as "Smitt" instead of "Smith," or even "Smith SB" instead of "Smith SE," may prevent accurate automated citation recognition and frequency calculations. It has been suggested that this type of mistake occurs in up to 10% of citations6. Interpretation of the impact factor must be done with an understanding of its true meaning, that it reflects the number of times, on average, that a source item in a particular journal is likely to be cited. This distinction between the rate of citation and journal quality can be easily illustrated by comparing journals listed in the annual Journal Citation Reports. According to the 2001 Journal Citation Reports, the American volume of The Journal of Bone and Joint Surgery had an impact factor of 2.138 and the journal International Orthopaedics had an impact factor of 0.427. Clearly, in the time-period under review (1999 through 2000), The Journal of Bone and Joint Surgery had, on average, approximately five times as many citations per article (source item) as International Orthopaedics. Does this, however, demonstrate that the scientific standard of The Journal of Bone and Joint Surgery or an article published in it was five times better than that of International Orthopaedics or one of its articles? While this can be debated, most would agree that this was not the case. In a more extreme example, the recorded impact factor distribution17 ranges from hundredths of a unit to >40. If the impact factor is assumed to be an indicator of quality, this suggests that some journals recorded in the Science Citation Index database are several thousand times better than others17. Clearly, while the use of impact factors to compare journals may be arguably possible on a qualitative level, it cannot be done on a quantitative basis.
It must be recognized that the Science Citation Index includes only approximately 5000 journals28 of an estimated world total of 126,00018,23; thus, it represents <4% of all journals. Journals not listed in the Science Citation Index database are often crudely referred to as having no impact factor (zero). This suggests, incorrectly, that 96%, or 121,000, of journals are never formally cited. The inclusion criteria for the index are also unclear, although it has been suggested that selection might be influenced by the commercial interests of the Institute12. Many of the journals listed have very low impact factors, suggesting that a high citation rate alone may not be a key consideration for inclusion. Citations made in journals not recorded in the Science Citation Index do not contribute to impact factor calculations12,29. Seglen reported that, within the field of mathematics, publications that were not included in the Science Citation Index database were cited more frequently than were publications that were included23. It is unclear whether such an anomaly exists in other fields. Review of the journals included in the Science Citation Index database has also shown an enormous bias toward those published in English3,9,18,23,30-32, with non-English-language journals given lower impact factors5,14. In discussing the results of a review of German social science periodicals, Seglen23 reported that only two of 542 such journals listed in a German database were included in the Science Citation Index. The United States3,23 and United Kingdom dominate medical publishing21,32 and hence the journals included in the Science Citation Index. It has been noted that the impact factors assigned to United States-based publications have been approximately 30% higher than those given to publications in the rest of the world, as a result of a higher frequency of self and colleague citation18,23. The international language bias extends further than simply the language of publication. English-speaking authors (who dominate the Science Citation Index) rarely cite non-English-language literature31, regardless of the otherwise implicit appropriateness of the citation itself. Several earlier publications have raised this concern and discussed its indirectly negative effect on the impact factor standing of non-English-language journals31,33,34. This problem has been compounded by the fact that authors from non-English-speaking environments seek publication of their work in journals published in English31. These authors may seek publication in a higher impact-factor-rated English-language journal rather than in what is perhaps a more appropriate non-English-language alternative. This further decreases the impact factors of these authors' native journals, making them less appealing to those for whom the journal impact factor is a selection criterion for seeking publication. Given the current pressure to conform to the impact factor, changing trends steering high-caliber authors away from seeking publication in their native non-English-language journals and toward larger, high-impact-factor journals are likely to contribute to the decline of many smaller journals that may have otherwise played an important role in the timely dissemination of knowledge. Differences in citation25 and referencing8,23 tendencies within individual fields limit the validity of cross-discipline comparison. For example, it has been reported that the mean number of references per article of biochemistry periodicals is three times that of mathematics periodicals8. Some fields encourage lengthy reference lists, whereas others dictate more concise or restricted bibliographic listings35. Because of this, Linardi et al.8 suggested that comparisons of journals on the basis of their impact factors should be limited solely to intra-area evaluation; they warned that inter-area comparisons may be both inappropriate and misleading. A wide range of other factors can also influence impact factor calculation. Ease of access to journals, publication immediacy, and type of publication material have all been identified as contributors to the impact factor. The availability of journals to authors and researchers can vary7. Theoretically, journals published more frequently8 may be more readily available for citation or may reduce publication lag. The fact that a journal or article is available electronically7,9 may also increase the rate of citation and thus the impact factor. As discussed above, only articles that are cited within two years after publication contribute to the impact factor3,17. The length of time that the peer-review process takes in moving articles toward publication1,21 may reduce the immediacy of the articles' impact. Many fields have a relatively long lag time to publication29, while others tend to move more quickly17,18,23,30,35. Use of a two-year reference period may not provide a complete picture of slower-moving fields1,29,35, and this is likely to cause bias against studies in clinical journals that require longer periods to complete the publication process (i.e., conceptualization, ethics approval, patient recruitment, treatment, follow-up, and write-up)17,35. The practice, by some journals, of allowing prerelease of details of so-called soon-to-be-published materials may allow citation before articles even "go to press," increasing the immediacy of impact (given the two-year time-frame). Historically, it has often taken several years before the data and concepts described in important articles have become of wider interest or have gained acceptance within the broader scientific community. This clearly reflects the short-coming of blind acceptance of the current two-year impact-factor data. For example, it took nearly half a century before the importance of the earlier work by Haboush36 and Wiltse et al.37, examining the use of polymethylmethacrylate in musculoskeletal implant fixation, was realized. Thus, use of the impact factor alone would have considerably underestimated the perceived medical and scientific importance of these authors' work. The type of research being reported can affect the journal impact factor because of citation limitations. Scientific articles tend to cite only scientific articles, whereas clinical articles cite both scientific and clinical articles9,23, thus allowing a much larger pool for citation. In a similar context, general journals tend to have higher impact factors than specialist journals13,25,35 because of the larger pool for citation. Finally, those who choose to use the impact factor as a measure of quality must recognize that the Institute of Scientific Information is a private for-profit company that enjoys an unchallenged monopoly on the market of citation-frequency recording12. Thus, despite the valuable contribution that this company has made to the scientific community, it does have a commercial interest in the development and application of its products3,5,12,13, which may not always align itself with pure academic intent.
The quality of an individual scientific research paper is an extremely difficult concept to define and quantify3,9,23,25. Comparing the quality of two separate articles (perhaps from different journals) has proven to be an even more difficult task. The frequency of citation has been adopted as a rough indicator of quality25. Although a high citation rate may not always be associated with high quality6, most citations in most papers are not refuted or discredited by the authors of the paper29. Thus, it is still widely accepted among authors that citation of work by others imparts a degree of prestige and professional recognition34. While impact factors may be useful for the qualitative evaluation of journals, the usefulness does not extend to individual articles. In fact, it has been reported that 50% of citations recorded in the Science Citation Index come from just 15% of articles published12,38 and that the most cited 50% of articles account for approximately 90% of citations3,18,23. Thus, the impact factor of a journal is likely to be largely influenced by a small percentage of its published articles17,30. Previous authors have suggested that there is virtually no correlation between the frequency of citation of an individual article and the impact factor of the publishing journal6,17,23,30. Similarly, it is important to note that the impact factor does not reflect the quality of the peer-review to which a journal subjects its articles9. Confirming the heterogeneity of article citation rates within an individual journal, previous studies have suggested that a large percentage of published articles are never cited6. Studies have reported rates of non-citation between 9% and 46%, depending on the publication discipline3,6,29,39. Schwartz40 suggested that up to 22% of all articles published in the international medical literature are not subsequently cited. Given the clear limitations of extrapolating the impact factor of a particular journal to the quality assessment of an individual article or author, it seems that such a practice would be limited and actively discouraged. This, however, has not been the case. Much has been written about the misuse of the impact factor in such matters as the determination of staff and academic appointments3,4; the allocation of staff promotions and tenure3-5,10,11,16,18,23,25,34,35; the awarding of research funding7,10,11,16,18,23,25,35, grants, scholarships, fellowships8,34, and research awards3; and even the evaluation of postgraduate courses8. The Institute of Scientific Information itself suggests that the primary utility of the Journal Citation Reports is to assist librarians and researchers in managing journal collections. In addressing the extension of this tool to academic evaluation, the Institute of Scientific Information states that, while the impact factor may provide a gross approximation of the prestige of journals, it does not advise using this value as the sole means of comparative evaluation. Misunderstanding of the impact factor and inappropriate weighting of its importance have affected the author-journal relationship, often greatly influencing authors' selection of the journals to which they submit their manuscripts5,7-9,18,23,25,35. Many authors may be tempted, or feel pressured, to select the highest impact-factor-rated journals likely to accept their article for publication while rejecting journals whose target audience may in fact be more suitable and receptive to the publication itself3,12,23,33,35.
The impact factor of a journal cannot be considered to represent the citation rate of an individual article6,20,23 and does not permit assessment of the quality of an individual article or author6,26,33. While the impact factor may, in certain circumstances, be a useful subjective tool for grading journal quality, it is not appropriate for quality assessment of individual articles or authors5. Individuals and governing bodies that use the impact factor for these purposes demonstrate a poor understanding of a tool that should perhaps more appropriately be termed the "journal citation ratio" or the "journal citation index." The inappropriate use of the impact factor may reflect the increasing pressure on such bodies to judge the quality of research and researchers and the decreasing time devoted to appropriate review of articles from publication portfolios. This point was raised by Figa Talamanca, who suggested that the impact factor "relieves faculty and administrators of the burden of reading and comparing papers."12 In essence, we have yet to discover an all-encompassing and universally accepted quantitative measurement of the quality of a journal, article, or author that can be confidently used to obviate the need for peer review and assessment by suitably qualified and experienced experts in the field23,38. As a very general rule, journals with exceptionally high impact factors are among those widely considered the most prestigious1,38, although this is a subjective qualitative measure and does not consistently hold true. Across the full range of journals listed in the Science Citation Index database, impact factor alone does not correlate well with author or researcher-defined journal quality41. Despite the recognition that the impact factor is an imperfect measure and forty-five years of criticism, there is no obvious alternative1-3,41. Thus, those forced to use this tool for direct journal comparison should be encouraged to remain open-minded and cautious8, with an awareness of the inherent limitations of its use7. While the index is arguably useful for qualitative intradiscipline comparison, application beyond this (i.e., to interdiscipline assessment) has little value (validity)42. Extension of journal-impact-factor data to individual articles and authors is inappropriate and should be avoided.
  CiteULike  Connotea  Del.icio.us  Facebook  Technorati  Twitter What's this?
This article has been cited by other articles:
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Top
Abstract
Introduction
| home | CME | help | feedback | subscriptions | archive | site index |
|
Stanford University Libraries' HighWire Press (TM) assists in the publication of JBJS Online. |