Centre for Science and Technology Studies Centre for Science and Technology Studies 2333AL Leiden Zuid Holland 31715273909
  • Blog
  • The Metric Tide almost six months on: Still riding the wave?

Blog archive

This is the archive of our old blog. Please visit our new Leiden Madtrics blog.

The Metric Tide almost six months on: Still riding the wave?



metric tideIn July 2015 a major report was published into the prospects for increased uses of metric indicators in the UK Research Excellence Framework. Alex Rushforth considers some of the subsequent events which may yet scupper the impact that the Report's findings go on to have.

Cast your mind back to July 2015 and those heady days of summer. Amongst other things, this month saw the publication of the Metric Tide, a large-scale independent report commissioned by HEFCE into the proposed uses of metric indicators in future Research Excellence Framework (REF) – the UK's nation-wide exercise for auditing university research. Co-authored by an appointed team of science policy experts including CWTS's Paul Wouters, the report recommended the use of 'responsible metrics' which could - where appropriate - support (but not replace) expert peer review. This message chimed with other recent statements articulating concerns about misuses of metrics and with calls for minimum standards emerging from the scientometrics community itself. Following recent discussion in science studies literature over Responsible Research and Innovation, the report laid out five basic principles of 'responsibility' any efforts to introduce metrics into the REF should adhere towards: robustness, humility, transparency, diversity, and reflexivity.

Since the summer a number of political developments have occurred in the UK which may yet stall such recommendations. So what has been happening and how might the report's message for responsible use of metrics be blown off course?

On its publication in July, the report's findings received something of a mixed response. As well as criticisms from some in the academic blogosphere, more importantly its recommendations appeared to run against prior wishes of the universities minister Jo Johnson, a figure well known to be keen on replacing peer review with metrics in the next REF. Recommendations not to replace peer judgments also ran counter to claims made by Elsevier, both earlier in 2015 and after the July publication, that the company could administer a reliable metrics-based REF at a fraction of the costs of the 2014 REF (the peer-review based assessment cost approximately £230 million to administer). Were such recommendations to be adopted, they would surely run counter to several of the Metric Tide's core criteria for responsibility. For instance, calls for humility, i.e. metrics should not replace peer review, are one obvious aspect such a move would discard. Likewise this would mean ignoring the report's concerns about robustness of data, namely that extant indicators and data infrastructures are simply not yet strong enough to support metric-driven evaluation across the board. And in terms of transparency, could commercial providers really be trusted to make their data collection and analyses open to public scrutiny?

Despite these concerns, since the Autumn the door for a metrics-based REF has been left open, as a whole raft of reviews into the governance and funding of higher education sector have been initiated, including the dismantlement of the REF's administering body, HEFCE and into the exercise itself.  Initial speculation about substantial 'efficiency' changes to the HE sector, were sparked back in September when Business minister Sajid Javid decided to hire management consultancy firm McKinsey and Co. to perform a whole-scale funding review of the Department for Business, Innovation, and Skills, the governmental department out of which universities are run. Although details of McKinsey's report have been kept closed, in light of substantial cuts across Whitehall departments, the continued funding of quango HEFCE and by extension the REF were very obvious candidates for reforms.

A Green Paper released on 6 November, authored by Jo Johnson announced plans to keep the REF, but confirmed the dismantlement of HEFCE and setting-up of the 'Office for Students' in its place. The Green Paper indicated the Metric Tide's conclusions over responsible metrics would be 'considered', but also stated 'we should champion better and more effective use of data and metrics'. In late November, Chancellor George Osborne (the minister in charge of economic and financial matters) announced in his autumn statement to parliament that recommendations made in an independent review of Research Councils (another area of the system being shaken-up) would be implemented, including an independent review of the REF. On Monday 7 December it was announced this review would be carried out by a 'senior figure', possibly a former vice-chancellor or someone from industry and be separate in scope from the Metric Tide report. A week later this figure was announced as Lord Stern, an academic economist and current president of the British Academy.

Although the date of the next REF has been signalled as 2021, in light of these tremors, what the exercise would look like in terms of methodology is not at all clear. By 16 January 2016 we may well know a bit more. This marks the deadline for submissions of follow-up questions and consultations to the Green Paper, offering various parties opportunity to give evidence and recommendations for Jo Johnson to consider before acting further on the Paper's proposals. At the start of November professor James Wilsdon announced that he had met with Jo Johnson to reiterate the importance of responsible metrics. The Metric Tide group also reconvened in late November to prepare its submission to the consultation. Speaking in Prague at the start of December, Wilsdon reiterated that the Metric Tide report is simply one of several information sources being submitted for consideration as part of the consultations.

With Jo Johnson seemingly still flirting with the idea of a more cost-efficient, metrics-based REF, and with and multi-national actors Elsevier and Thomson Reuters circulating, there must surely be concerns as to whether any calls for responsible metrics will be able to shout loudly enough over the coming months. The REF has long been a trailblazer in terms of large-scale assessment exercises and irresponsible insertion of metrics into this process could have knock-on consequences into the credibility and legitimacy of evaluative bibliometrics further afield than the UK. Despite these uncertainties, Wilsdon himself remains cautiously optimistic about the forthcoming review processes, and was personally enthusiastic about the appointment of Lord Stern to head the independent review of the REF. In a personal communication about the Stern appointment, he remarked: “I’m sure he will take a balanced, evidence-informed look at the issues, including drawing on The Metric Tide. And his role as President of the British Academy means that he has a rich understanding of the issues facing the arts, humanities and social sciences with respect to research assessment, and particularly any moves to a metrics-based system.” Whilst the ultimate fate of the Metric Tide's message may still be in the balance, Wilsdon strikes an optimistic note that there is still everything to play for. After the January consultations we will have to wait for July 2016 to discover the outcomes of Stern's report. Whether the Metric Tide can ride these waves, towards what it hopes will be the smoother waters of responsible metrics-informed evaluations, remains to be seen. 


About Alex Rushforth

Assistant professor. Alex works in the social studies of science, specializing in research evaluation, uses of indicators and research assessment reforms. 


No comments

Mandatory fields
    Share on:
    Subscribe to:
    Build on Applepie CMS by Waltman Development