Reflecting on the Stern Review

Published: Posted on

On 28th July 2016, Lord Stern published his much anticipated review of the REF: Building on Success and Learning from Experience.  The report neither proposed a wholesale move to metrics to assess research quality nor the abolition of REF – both apocalyptic scenarios trailed early in the review process.  Rather, the proposals for change are thoughtful and largely incremental, seeking, as the title of the report suggests, to build on the positives of REF and address some of its problems.

For those of us who have been around a while, watching the evolution of RAE and REF, this all had a bit of a familiar ring: after the exhausting process of undertaking yet another round of research assessment, academia and government both raise familiar complaints about the REF (cost, burden, robustness of the review process) and a review is set up to address these issues.  Proponents of metrics argue that all we need to do is look at research income and citations data, and we will have the answer.  Others point out the deficiencies of peer review, and the huge burden on REF panels in reviewing all submitted material and reaching appropriate judgements.  This time round we had the added bonus of Impact to take aim at.  Then as the realities of the alternatives begin to bite, and review panels read submissions from HEIs who have begun to think about how changes might actually affect them, everything simmers down and the report, when it appears, tinkers around with the process – and generally adds yet more to the burden of putting the submission together for HEIs, and to the assessment burden for the sub-panels.

So, was Stern any different?  Well, a lot depends on how the twelve recommendations are actually taken forward, and we will have to wait for the technical consultation to see what some of the proposals might actually mean in practice.

The REF team at HEFCE will have an interesting challenge coming up with an interpretation of “research-active staff”

For example, for the first time, the report seeks to address the real impact on individual academics of not being returned to the REF, by proposing that all “research-active staff” should be submitted (with accompanying recommendations about the submission of outputs aimed at lessening the burden of this proposal on institutions and sub-panels) .  But what does this mean?  Those of us who had to try to determine how to interpret and apply the definition of “independent researcher” in our REF 2014 submissions may feel a certain sinking of the heart at this point.  As it happens, defining what constitutes “research-active” is very challenging – possibly why the Stern report didn’t attempt to do so.   Different institutions have different employment patterns, and different disciplines have different understandings of at what point an individual is genuinely independently research active.  It may not even be straightforward to say all staff on three-legged contracts should be returned: is this a model that is fair to post-92s where the research culture is still evolving?  What about staff on junior post-docs?  In STEM subjects, these are usually staff working under the direction of a Principal Investigator, not undertaking their own autonomous research, but this may not be the case in other disciplines.  The REF team at HEFCE will have an interesting challenge coming up with an interpretation of “research-active staff” for the consultation which takes into account these differences but does not completely lose the spirit of the Stern proposal.

Other proposals are similarly challenging, although not necessarily in the same way.  An institutional level environment statement, complementing UOA level statements, may seem straightforward, but how do we avoid the situation where the answer to the definition of an excellent research environment is Oxford or Cambridge?  What about high quality monotechnics, or indeed post 92 institutions trying to do the very best by their research-active staff, but lacking the resources available to our oldest foundations or, perhaps, the benefits of being located in London?

what is needed with impact is more confidence within some of the sub-panels to review types of impact less common in their disciplines

Similarly with Impact: broadening the guidance on what constitutes Impact may seem to address the concerns of institutions and researchers, but actually what is proposed only adds one element that was not included in the REF2014 definition (“academic impacts outside the field”); what is needed with impact is more confidence within some of the sub-panels to review types of impact less common in their disciplines.  Here, more dialogue  between the disciplines, and particularly between sub-panels, could be hugely beneficial.  Social Science sub-panels understood how to review policy impact, even if others did not.

Where does this all leave us?  For me, a valuable element of the Stern review, and what sets it apart from its predecessors, is the regular refrain that the burden on sub-panels (and hence also on institutions) should not be increased.  The recommendations taken collectively – notwithstanding the kinds of issues discussed above – do offer a way forward that attempts to balance change with not increasing burden.   This has to be welcomed.  They also attempt to address some of the more unfortunate behaviours that have developed as RAE/REF has evolved over time.  I don’t believe that game-playing can ever be wholly removed from a process such as this, with it high stakes and complex rules of engagement, but I can see the degree of latitude for some behaviours being reduced, hopefully to the benefit of those academics for whom this is not a game, but a judgement on their working life.

To return to the question, was Stern different?

To return to the question, was Stern different?  Yes, I think so.  The analysis of issues is careful and perceptive, and the recommendations attempt to address well-recognised problems, without just adding to everyone’s workloads.  If the precise formulation of how to take the recommendations forward is difficult, that should not deter us from trying to do our best, collectively, to come up with a way forward which does indeed identify and foster excellence, wherever it may be found.

Elizabeth is Deputy Director – Research Planning and is responsible for managing the Research Planning Team, overseeing REF preparation, and especially support for Impact.  She also co-ordinates support for Open Access and Open Data working closely with colleagues from across Professional Services, but especially ITS, Library Services and RIS,  and is business owner for the institution’s CRIS system, Pure.  She is secretary to the University’s Research Committee and works closely with the PVC (Research and Knowledge Transfer) to support delivery of the University’s Research and especially REF objectives.

Leave a Reply

Your email address will not be published.