Ratings and rankings: Judging the evaluation of quality
Article Outline
Marion E. Broome, PhD, RN, FAAN
At a round table there is no dispute about space… .
An Italian Proverb
Ratings and rankings are very important in several areas of nursing. In nursing practice, the award of Magnet status is viewed as a clear reflection of excellence in nursing and patient care.1 Health systems are very proud of this accomplishment and use the designation as a marketing tool to attract physicians, patients, and nurses. Nursing schools rely heavily on the U.S. News and World Report rankings for graduate programs,2 as well as their ranking according to the amount of NIH funding they secured the previous year,3 to attract high quality students and faculty.Another ranking system that is receiving more attention lately is the “journal impact factor” in Journal Citation Reports (JCR) published by the Institute for Scientific Information (ISI).4 In the past, these citation reports were used primarily by librarians as a bibliometric indicator and by publishers when making business decisions based on the competitiveness of a journal. Now they are also viewed by editors as an effective tool to assess the standing (ranking) of their journal with similar products, and authors who attempt to determine what journal will provide the most “prestige” when their paper is published. And increasingly, university appointment, promotion and tenure committees use the journal citation reports and journal impact factors as an objective method to measure the “quality” of the journal. They require candidates for promotion/tenure to indicate the impact factor for each journal in which they have published to better assess the rigor of their scholarship. This ranking is presumably a highly objective and quantitative measure that many academic institutions believe they can put faith in as they make important decisions about an individual’s career. However, this ranking must be carefully considered and applied to be fair to all involved and effective in assessing the quality of journals.
The Journal Citation Reports cover more than 7500 journals published around the world.4 These reports provide information for each journal on the following parameters: impact factor, immediacy, total articles, total citations, and cited half-life. Impact factor is calculated by dividing the number of current citations a journal receives (in say 2002) to articles published in the 2 previous years (in 2000 and 2001) by the number of articles published in those same years. More simply, the impact factor can be thought of as the number of citations the average article receives per year during the two years after the year of publication.5 The cited half-life is a measure of how long articles in a journal continue to be cited after publication. Logic then supposes that the higher the quality of an article (ie, innovation, significance, rigor, etc), the longer it will be cited after it is published.
So why should review committees be cautious about using the impact factor to assess the quality of a publication? There are two primary reasons: sociological and statistical factors that influence the variability (hence instability) of the impact factors. Sociological factors include the type of journal and subject area, whereas statistical factors include the size of the journal and the citation measurement window.5 Nursing impact factors, for instance, are known to be relatively small (less than 2), however so are journals in mathematics and materials science and engineering,5 when compared with journals in fundamental life sciences and clinical medicine. The number of authors (authors have a tendency to cite themselves at higher rates than other authors cite them), frequency of publication (frequent publications tend to have higher impact factors), size of the journal (eg, number of papers and references), type of papers (eg, journals that publish review papers have a higher impact factor), and whether it is published in English all influence the size of and variability of the impact factors each year.5, 6 Finally the subject area can also have a substantial influence. Currently only 31 nursing titles are indexed and most of these are nonspecialty journals such as Nursing Outlook, Nursing Research, and Journal of Professional Nursing. And although as editor of Nursing Outlook I was proud of our rating of 3 of the 31 journals rated in 2003, knowing how variable the impact factor can be year to year, I temper my enthusiasm and look forward to some trending over several years. And I would encourage anyone who sits on a promotion and tenure review committee to do the same. They should consider several factors that reflect the quality of a journal, including the stature and experience of the editor and members of the editorial board, rejection rate, other professionals’ opinions, time from acceptance to publication and, of course, the actual quality of the authors’ papers. Like most items used in evaluation there is always some degree of subjectivity and complexity associated with criteria that need to be appreciated by those applying them. Impact factors deserve no less scrutiny.
References
- . Advancing the Magnet Recognition Program in master’s education through service learning . Nurs Outlook . 2004;52:134–141
- America’s best graduate schools . U.S. News and World Report . 2006; Special issue. edition
- National Institutes of Health. Award Trends. Available at: http://www.grants.nih.gov/grants/award/awardtr.htm. Accessed August 21, 2005.
- Thomson ISI. Welcome page. Available at: www.isinet.com. Accessed August 11, 2005.
- . Impact factors (use and abuse) . In: Perspectives in Publishing . Philadelphia (PA): Elsevier Science; 2000;p. 2–6
- Garfield E. The use of journal impact factors and citation analysis for evaluation of science. http://www.garfield.library.upenn.edu. Accessed August 11, 2005.
PII: S0029-6554(05)00169-7
doi:10.1016/j.outlook.2005.08.002
© 2005 Mosby, Inc. All rights reserved.
