Centre for Science and Technology Studies Centre for Science and Technology Studies 2333AL Leiden Zuid Holland 31715273909
  • Blog
  • The importance of taking a clear position in the impact factor debate

Blog archive

This is the archive of our old blog. Please visit our new Leiden Madtrics blog.

The importance of taking a clear position in the impact factor debate



Blog post IF smallLast week, the paper A simple proposal for the publication of journal citation distributions  appeared online. The first author is Vincent Larivière, a bibliometrician from Université de Montréal, and the last author is Stephen Curry, a biologist from Imperial College and one of the authors of the Metric Tide report  published last year by the Higher Education Funding Council for England. Importantly, the paper is co-authored by representatives of a number of prominent journals (Nature, Science, EMBO Journal, and eLife), a publisher (PLOS), and the Royal Society in the UK.

Larivière et al. argue that, when journals report their impact factor (IF), they should also present the underlying distribution of citations over papers. This will draw awareness to the strong skewness of citation distributions and will show that the IF of a journal is determined largely by a small number of highly cited papers.

The idea of complementing IFs with information about the underlying citation distributions is attractive and makes sense to me. I also agree with many of the comments made by Larivière et al. on technical weaknesses of the IF. In this blog post, however, my focus will be on another element of the paper by Larivière et al., namely the distinction made by the authors between the use of IFs at the level of journals and at the level of individual papers.

Journal-level vs. paper-level use of impact factors

Larivière et al. argue that “research assessment needs to focus on papers rather than journals” and that the IF “is an inappropriate indicator for the evaluation of research or researchers”. On the other hand, Larivière et al. also state that they “are not arguing that the journal IF has no value in the comparison of journals”. Hence, according to Larivière et al., IFs can be used for making comparisons between journals, but not for making comparisons between individual papers and their authors.

The question I want to raise in this blog post is what this distinction between the use of IFs at the level of journals and at the level of individual papers actually means. To me it appears difficult to separate these two ways of using IFs. When one uses IFs to compare journals, it seems that usually in one way or another one does so in order to compare papers in the journals. My point is that everyone involved in discussions on the IF needs take a clear and consistent position. In particular, critics of the IF need to accept the full consequences of their criticism. Either they need to fully reject any use of IFs or they need to explain very carefully which uses of IFs should be rejected and which ones are acceptable.

To illustrate my argument, I will provide two examples of the interrelatedness of the use of IFs at the level of journals and at the level of individual papers.

Example 1: Using journals to filter the scientific literature

Suppose a researcher has found two papers that both seem interesting to read, but he has time to read only one of them. The researcher has skimmed both papers, and based on this the two papers seem equally attractive to read. Suppose also that one paper has appeared in a journal that is known to carefully select the work it publishes and to have high quality standards, while the other paper has appeared in a journal that is known to have low standards. Since the researcher doesn’t have any other information that helps to choose which paper to read, the researcher chooses to read the paper in the selective high-quality journal. Based on his knowledge of the two journals, the researcher assumes that the paper in this journal will probably be of more interest than the paper in the other journal.

Approaches like this one to filter the scientific literature seem to be used by almost all researchers. But how do researchers know which journals have strict acceptance criteria and a high level of quality control and which ones adopt lower quality standards? Partly researchers know this from their experience as a reader, author, and referee of journals, and partly they know this because colleagues informally share their experiences as readers, authors, and referees. In addition, researchers may also use quantitative indicators such as the IF to get an impression of a journal’s quality level. This seems to be accepted by Larivière et al., since they accept the use of IFs to make comparisons between journals. However, in our example, the comparison that is made between two journals, either based on personal experience or based on IFs, is used to choose which paper to read. Hence, journal-related information, such as IFs, is used not only to compare journals but also, in an indirect way, to compare papers.

Example 2: Using journals to compare researchers

Suppose a researcher has two colleagues who are each working on their own research project, and suppose each of the two colleagues invites the researcher to join their project. The two projects look equally interesting, but the researcher has limited time available and therefore is able to join only one of the two projects. How to choose which project to join? Without any further information that helps to make a choice, the researcher may decide to look at the past performance of the two colleagues and to choose to collaborate with the colleague who seems to be the more capable researcher.

Determining in an accurate way whether one researcher is performing better than another will often take quite some effort. However, many researchers need to make decisions on whom they want to collaborate with almost on a daily basis, and therefore they need to have a quick way to get at least a rough impression of someone’s capabilities. Researchers may then rely on easily available information, such as the number of papers someone has written during the past years, the journals in which these papers have appeared, and someone’s h-index. Most researchers are probably aware that this information offers only a rough impression of someone’s performance, but for some decisions they may consider it sufficient to rely on such a rough impression.

Suppose the researcher in our example decides to compare his two colleagues by comparing the journals in which they have published. Like in example 1, the researcher will probably take into account his personal experience with the journals, but in addition he may also use quantitative indicators such as IFs. IFs are then used to compare journals and, in an indirect way, to compare researchers.

Taking a clear position

In the above two examples, IFs are used not only to compare journals but also, in an indirect way, to compare individual papers and their authors. Larivière et al. reject the use of IFs to compare papers and their authors, and therefore I expect them to also reject the use of IFs in the above two examples. However, when Larivière et al. state that they “are not arguing that the journal IF has no value in the comparison of journals”, what kind of use of IFs to compare journals do they have in mind? There is a strong interrelatedness of the use of IFs at the level of journals and at the level of individual papers. A simple statement that IFs can be used at the former level but not at the latter one therefore doesn’t seem satisfactory to me.

In discussions on IFs, there are two clear positions that one could defend. On the one hand, one could take the position that in certain cases the use of IFs at the level of journals and at the level of individual papers is acceptable. On the other hand, one could take the position that any use of IFs should be rejected. Although these positions are opposite to each other, they each seem to be internally consistent. Larivière et al. take neither of these positions. They argue that the use of IFs at the level of individual papers should be rejected, while the use of IFs at the level of journals is acceptable. This seems an ambiguous compromise. Given the strong interrelatedness of the use of IFs at the two levels, I doubt the consistency of rejecting the use of IFs at one level and accepting it at the other level.

Conclusion

Discussions on impact factors involve many different types of arguments and therefore have a high level of complexity. To have a fruitful debate on the IF, it is essential that everyone involved takes a position that is clear and internally consistent. Critics of the IF, such as Larivière et al. but also for instance the supporters of the San Francisco Declaration on Research Assessment , need to accept the full consequences of their criticism. Rejecting the use of IFs at the level of individual papers seems to imply that there also is little room for the use of IFs at the level of journals. In order to be consistent, critics of the IF may even need to further extend their criticism. If one rejects the use of IFs because of the variability in the quality of the papers in a journal, this calls into question whether other types of information on the quality level of journals, such as researchers’ personal experiences with journals, can still be used. Shouldn’t the use of this information be rejected as well? For instance, when deciding which paper to read or which colleague to collaborate with, shouldn’t one completely ignore any information, both from personal experience and from quantitative indicators, on the quality level of the journals in which papers have appeared? This may seem quite an extreme idea, but at least it represents a clear and consistent position, and in fact some have already taken concrete steps to move in this direction.

 

I would like to thank Sarah de Rijcke, Philippe Mongeon, Cassidy Sugimoto, and Paul Wouters for valuable feedback on a draft version of this blog post.

 

Below some other publications by CWTS staff members and co-workers on topics closely related to the above blog post are listed:

Blog post by Sarah de Rijcke criticizing too simplistic notions of ‘misuse’ and ‘unintended effects’ in debates on the IF.

Blog post by Philippe Mongeon, Ludo Waltman, and Sarah de Rijcke on journal citation cartels.

Blog post by Paul Wouters on the San Francisco Declaration on Research Assessment.

Recent research article by Alex Rushforth and Sarah de Rijcke demonstrating how biomedical researchers increasingly use the IF in quite routine knowledge making activities. The article concludes that ‘misuse’ and ‘unintended effects’ are far too simplistic as analytical concepts and offers a new conceptual framework for understanding fundamental effects of the IF on research practices.

Classical research article by Henk Moed and Thed van Leeuwen on inaccuracies in the IF due to technical inconsistencies. A short correspondence on this topic was published in Nature.


About Ludo Waltman

Ludo Waltman is professor of Quantitative Science Studies and scientific director at the Centre for Science and Technology Studies (CWTS) at Leiden University. He is a coordinator of the Information & Openness focal area and a member of the Evaluation & Culture focal area. Ludo is co-chair of the Research on Research Institute (RoRI).


3 comments

Mandatory fields
  • Fei Shu July 13th, 2016 10:09 pm
    Let’s see a similar scenario using the world university rankings (QS, Times, Leiden, etc.) at the level of universities and the level of students. Parents would like to send their kids to top universities that are ranked by the world university rankings; but it does not mean that all students in top universities are superior to students in other universities. We cannot use the world university ranking to differentiate university studetns or graduates. For example, two university graduates apply for the same position; both applicants are excellent based on their similar CVs; one is from Harvard while another is from a state university. It may be easy for HRs to select the Harvard graduate because of the big name (from the ranking) but we cannot determine that the Harvard graduate is more excellent than another one. Using the world university ranking in this selection process is very convenience but not fair to evaluate individual’s competency and potential success. Also, scholars intend to publish their papers in high-IF journals (IF is almost the only criteria for Chinese scholars deciding where to publish), but not all papers in high-IF journals are superior to those in low-IF journals. If the purpose is to differentiate individual papers in a short term, the use of IFs is a convenience way; Otherwise, it is not appropriate to use IFs to evaluate individual’s research performance.
    Reply
  • Stephen Curry July 13th, 2016 10:06 am
    My first reaction is that your two cases are far removed from the real life experiences of researchers. I have never faced choices about what to read or who to collaborate with on these bases. Our focus was very much on research assessment in terms of hiring, promotion etc and in those terms, deflecting some attention onto the distributions seems like a positive move. 
    It is true that we did include statements suggesting that JIFs could (if the differences were great enough) indicate significant differences between average journal performance. This is a fairly straight-forward observation based on the data. But I don’t think it necessarily follows that our stance is contradictory – since the main thrust of our paper is that people should not rely unduly on simple metrics (JIFs or citations) when evaluating research. 
    Reply
  • David Colquhoun July 11th, 2016 9:41 pm
    You say "Approaches like this one to filter the scientific literature seem to be used by almost all researchers". That's odd because neither I nor nay of my friends has ever done anything of the sort. I suspect that this is just one of many myths that bibliometricians believe about how science works.
    It beats me why anyone is still arguing about impact factors. It's almost 20 years since Seglen showed what an impossibly bad way they are to evaluate individuals. (Working scientists already knew that, of course, but it was nice to have some numbers).
    It is a disservice to science to perpetually discuss the matter as though there were any controversy about it.
    The only way to tell is a paper is any good is to read it and understand it. That is what I have always done when selecting people for jobs. By pretending otherwise, the bibliometrics industry is harming and corrupting science. Please stop it!
    Reply
Share on:
Subscribe to:
Build on Applepie CMS by Waltman Development