• Blog
  • Let's move beyond too simplistic notions of 'misuse' and 'unintended effects' in debates on the JIF

Let's move beyond too simplistic notions of 'misuse' and 'unintended effects' in debates on the JIF



pecking orderResearchers in the field of scientometrics are quite often robust in their criticisms. Last week, the paper A simple proposal for the publication of journal citation distributions was published online. The contribution by Lariviere et al. (2016) takes a familiar normative stance regarding the 'misuses' and ‘unintended effects’ of indicators, and in particular the JIF. Arguments against the JIF often cite its technical shortcomings, for instance in claiming it is open to manipulation and misuse by editors and uncritical parties. Larivière et al. (2016) argue that the IF "is an inappropriate indicator for the evaluation of research or researchers" because the single numbers conceal "the full extent of the skew of distributions and variation in citations received by published papers that is characteristic of all scientific journals" and therefore "assume for themselves unwarranted precision and significance.” The authors hope that the method they propose for generating the citation distributions that underlie JIFs will "help to refocus attention on individual pieces of work and counter the inappropriate usage of JIFs during the process of research assessment."

Not merely for the sake of argument

Improving validity, reliability and transparency are obviously useful endeavors, but I will argue that it is more helpful if we do not take received assumptions of the JIF as given. Instead, we should position as a focus of intervention the multiple roles and influences of the JIF in actual research practices. My primary concern is this: by limiting solutions to ‘improper’ indicator uses to questions of validity or transparency, we assume that more transparency or better indicators will necessarily give rise to better evaluation practices. Though I applaud sincere, methodologically sophisticated calls for more transparency such as the one made by Larivière et al., I am afraid they do not suffice. The recourse we then take is towards an upstream solution, guided by an optimistic yet also slightly technocratic mode of 'implementation' (De Rijcke & Rushforth, 2015). If journals would indeed start to publish the citation distributions behind their JIFs, what exactly would this change on the shop-floor, in assessment situations, and in the daily work of doing research?

The JIF trickles back up

A general criticism that can be made about the well-known accounts against the JIF is their inattention towards the ‘folk theories’ of the JIF as applied by scientists and evaluators in actual practices (Rushforth & De Rijcke, 2015; see also Aksnes & Rip, 2009). “What characterizes folk theories is that they provide orientation for future action… They are a form of expectations, based in some experience, but not necessarily systematically checked. Their robustness derives from their being generally accepted, and thus part of a repertoire current in a group or in our culture more generally.” (Rip 2006, 349) In our research in biomedicine we for instance found that researchers use these ‘folk theories’ to navigate quite routine knowledge making activities, including selecting useful information from the overwhelming amounts of literature they could potentially read; settling discussions over whom to collaborate with and when; and how much - additional - time to spend in the laboratory producing that data. Given the extent of the embeddedness, I must say I feel ambivalent about statements that the JIF ‘misleads’. For one, not all of these embedded uses are grounded in naïve assumptions about the citation performance of individual papers in particular journals with a certain JIF. Secondly, who misleads whom? These different embedded uses of the JIF will ‘trickle back up’ into formal assessment procedures, because “[a]uditors are not aliens. They are versions of ourselves.” (Strathern 1997, 319)

An example

Let's consider the hypothetical situation of a formal assessment procedure in which a research group in oncology is looking to hire a new professor. The hiring committee has at its disposal the publication lists of the candidates. These lists also specify the JIF of the journals in which the candidates have published. Now suppose that the committee takes a look at the publication list of one of the candidates. The committee members start to compare the journals on the list by way of the JIFs. They see that this researcher mainly publishes in the top-tier journals in oncology. If we take the warning of Lariviere et al. to heart, we would advise the committee members not to conflate these numbers with the actual citation impact of the individual papers themselves. And rightly so. But does this mean that all uses of the JIF in this formal hiring procedure in this particular setting are off limits? Larivière et al. should answer this question with a 'yes', seeing that they disapprove of all uses of the JIF in the assessment of individual researchers. However, I think that in this case it is very well possible to come up with a reasonable motive for using JIFs to support the decision-making process about who (not) to hire. In some biomedical fields, different 'tiers' of journals with certain JIF-ranges can both denote a certain standing in a field and a particular type of scientific work (e.g. descriptive in the lower IF ranges versus causal in the higher IF ranges). So what a committee can do hypothetically in this field, on the basis of the JIFs, is assess whether a researcher mainly does descriptive work or primarily publishes about biological mechanism. In other words, the committee can make a substantive assessment of the type of work the candidate is involved in by looking at the JIF values of the journals in which she publishes (among other characteristics of the journal). And they can use this information, for instance, to deduce whether or not the candidate’s research lines would fit into the rest of the research team they are looking to hire for. The reader of this blogpost will understand that this is merely a hypothetical example. But I hope the point is clear: the JIF can acquire a range of different meanings in actual research and assessment practices.

Conclusion

Larivière et al. put forth a methodologically driven plea to focus not on the JIF but on individual papers and their actual citation impact. Though commendable, I think this strategy obscures a much more fundamental issue about effects of the JIF on the daily work of researchers and evaluators. JIF-considerations have a tendency to either move to the background other measures of scientific quality (e.g. originality, long‐term scientific progress, societal relevance), or to allow them to become redefined through their relations to the JIF and other quantitative performance indicators. In my opinion this insight leads to a crucial shift in perspective. For truly successful interventions into indicator-based assessment practices to happen, I think we need to move beyond too simplistic entry points to the debate of ‘misuse’ and ‘unintended effects’. My hypothesis is that researchers (and evaluators) continue to use the JIF in assessment contexts - despite the technical shortcomings – for the complicated reason that the indicator is already so engrained in different knowledge producing activities in different fields. Our research findings suggest that in calling for researchers and evaluators to ‘drop’ the JIF, people are actually calling for quite fundamental transformations in how scientific knowledge is currently manufactured in certain fields. This transformation is the primary, and also the quite daunting, task.


I would like to thank Alex Rushforth for collaborating with me on the project in biomedicine that I draw on extensively above. The text is partly based on our joint articles that came out of the project.

I would like to thank Paul Wouters and Ludo Waltman for valuable discussions that informed the preparations for this blogpost.

See also the blogpost by Ludo Waltman in response to the same article.

 

References

  • Aksnes, D. W., and A. Rip. (2009). Researchers' perceptions of citations. Research Policy, 38 (6), 895-905.
  • Rip, A. (2006). Folk theories of nanotechnologists. Science as Culture, 15 (4), 349-365.
  • Rushforth, A.D. & de Rijcke, S. (2015).  Accounting for impact? The Journal Impact Factor and the making of biomedical research in the Netherlands. Minerva, 53, 117-139.
  • de Rijcke, S. & Rushforth, A.D. (2015). To intervene, or not to intervene, is that the question? On the role of scientometrics in research evaluation. Journal of the Association for Information Science and Technology, 66 (9), 1954-1958.

 


About Sarah de Rijcke

Associate professor and deputy director of CWTS, and coordinator of the Science and Evaluation Studies research group. Her research focuses on the growing use of assessment procedures and bibliometric indicators in scientific and scholarly research, and the effects on knowledge production.


4 comments

Mandatory fields
  • Alex Rushforth July 20th, 2016 11:08 am
    Sorry I'm a little late to this debate - the system lost my original text. So here goes again...
    The argument of Larivière et al’s paper which started this debate seems to rest on a kind of path dependency- account – the scientific system is locked-in around an inferior tool when there are better alternatives available. Although I would agree with this, I also think path dependency accounts are not the only way forward in these kinds of debates –for my money they can be a bit too restrictive, as by suggesting salvation lies in adopting a superior tool they close-off a lot of what is going on in the research system which we should pay closer attention to. What I think is more important is to understand the kinds of conditions under which something like the impact factor can come to have such an big effect on the way research is conducted and governed – in interviews Sarah and I conducted with scientists this revolved around issues like scarcity of resources, excessive quantities of scientific literature from which to choose what to read (and cite), fiercely competitive job markets, the fact researchers are incentivized to write and not to read etc. It is confronting these sorts of issues (which studying the uses of the impact factor points us toward) more than technical limitations of the impact factor per se which I think would help produce better systems of evaluation and ultimately better science. I fear that getting journals to publish distributions alongside JIFs will not loosen the grip of the impact factor – it’s not that researchers we spoke to aren’t aware of limitations in how the indicator is calculated – it’s more they recognize it’s the de facto standard against which their prospect for external grants or a job interview will depend. It’s these latter kinds of issues which will need to change first if the impact factor is to go away.
    Reply
  • Sabrina Petersohn July 18th, 2016 2:13 pm
    Sarah, in line with our recent conversations I take it this is an illustration of how to "embrace" the JIF and h-index in the form of an extended research agenda. Two thoughts pop up: Studying the way the JIF has become embedded in research and assessment practices opens up a perspective on the positions of researchers as active users of metrics and not simply as evaluated subjects. This means probably stating the obvious here but it is a very interesting position to consider in terms of the relationships between indicator developers and users. Also the issue of the dependency on and adequacy in specific usage contexts of indicators comes up. Thanks for the food for thought, Sabrina
    Reply
  • Mark Patterson July 18th, 2016 11:15 am
    As another author on the paper in question, I’d like to add that I agree completely with your point towards the end that the JIF is "so engrained in different knowledge producing activities in different fields”. Many people have been *talking* about this thorny problem for years now.
    I was pleased to participate in this collaboration because I think it sends a signal that journals are now prepared to do something about this as well. Recently the ASM Journals went further by announcing that they would no longer promote impact factors on their websites (https://www.asm.org/index.php/asm-newsroom2/press-releases/94299-asm-media-advisory-asm-no-longer-supports-impact-factors-for-its-journals). Neither do eLife or PLOS.
    We’re not going to change things overnight, but for journals that do promote impact factors, sharing the citation distribution underpinning the impact factor is another way to point out that using the impact factor is a poor way to evaluate any individual article or scientist. And to reinforce Stephen’s point, we are not arguing that we should move to a situation where we only look at citations to articles instead. Practices in research evaluation have to change. There are some examples on the DORA website - http://www.ascb.org/inspiration-and-good-practices/.
    Reply
  • Stephen Curry July 14th, 2016 11:44 am
    It’s interesting – and stimulating – to see how different people read the same work. Let me give a brief initial reaction (because I am still digesting) to a couple of points.
    First I find your hypothetical situation a little odd (and as with Ludo Walkman’s scenarios), not very congruent with my own experience. To be honest I am astonished by the assertion in regard to biomedical research that: “what a committee can do hypothetically in this field, on the basis of the JIFs, is assess whether a researcher mainly does descriptive work or primarily publishes about biological mechanism.” This strikes me as an indefensible use of JIFs.
    Second, perhaps I need to re-read my own paper, but I did not intend to shift the focus to the “citation impact” of individual papers, though clearly we wanted people to move beyond simplistic interpretations of JIFs and focus on the content of papers. Indeed our closing paragraph states, “However, we hope that the broader message is clear: research assessment needs to focus on papers rather than journals, keeping in mind that downloads and citation counts cannot be considered as reliable proxies of the quality of an individual piece of research.”
    To my mind the aim of our work was not to eliminate impact factors but to increase awareness of the data that underlies them (and the real variation within it). There may well be nuances that we have overlooked or perhaps underplayed – but I shall continue to reflect.
    Reply
Share on:
Subscribe to:
Build on Applepie CMS by Waltman Development