On 6th February 2018, the HEFCE REF Team convened a meeting to discuss issues associated with the assessment of how impact derived from Public Engagement with Research (PER). Attendees included a mix of public engagement staff (both academic and professional services), sub-panel chairs, advisers and secretaries, and institutional REF people (again, both academic and professional services). It was chaired by Stephen Hill (HEFCE) and presentations were made by Paul Manners and Sophie Duncan of the NCCPE and three of the four Main Panel Chairs (Dinah Birch was not able to be present). Following this, a couple of questions were raised for small group discussion, focussed on the barriers to submitting PER based impact cases studies, and how the criteria for REF2021 might help to break these down.
The meeting was heavily attended and attendees were highly engaged, so it would be impossible to summarise everything that was said. Some of the points which jumped out at me in the course of the morning were as follows:
- There was unanimous agreement that Public Engagement is not itself impact: it is one of a number of mechanisms a researcher might use to lead to impact.
- The reservations that institutions had about submitting PER-based impact seemed to be caused by a lack of understanding of what high quality might look like, and concerns about how such impact might be evidenced. This was unpacked as relating to the challenge of evidencing what was often change to the way individuals or people thought, understood something, behaved etc, rather than something more concrete and hence easier to substantiate.
- This led to a discussion about impact intention. If an impact lead was clear about what change s/he was trying to achieve through their actions, then it ought to be possible to establish mechanisms to evaluate that change, even when this related to something intangible. Where impact intention was not properly thought through, evidencing was much more of a challenge. This highlighted the value of thinking through pathways to impact/impact intention at the outset of a research project.
- Some attendees argued that the panel criteria should be clear about what constituted appropriate evidence for different types and levels of impact; others asked for worked examples of 2*, 3* and 4* impact cases. This was balanced by a number of the sub-panel chairs arguing that providing a range of examples or too descriptive a set of criteria tended to lead to institutions feeling constrained to submit only the material that fitted what was laid out in the criteria. They were in favour of a more open and permissive approach
- Underpinning research was discussed, in the context of an earlier workshop the REF team had convened about the new concept of “body of work”. It was suggested that the terminology “accompanying research” might be better as this was less linear in its implications.
- A typology produced by the NCCPE to prompt discussion was seen as valuable, although needing more work. It identified three broad pathways to PER based impact, which were labelled Understanding (exploring meaning and values); Capability (enhancing skills, behaviour and networks) and Innovation (improving policies and practice and the way the world works). Alongside this was an interesting example of how a type of “intangible impact” (social capital) might be expressed and evidenced. It was felt that this set of materials, expanded to include more on evidencing other types of impact, and with further work to reflect some of the complexities of the delivery of impact, might usefully sit alongside the panel criteria to assist HEIs in thinking through, presenting and evidencing their PER based impact.
- As well as addressing issues relating to written guidance, it was important for sub-panel chairs and others associated with the process to make clear through their contacts with the sector that in just the same way there is no hierarchy of output types when it comes to grading output quality, nor is there – and nor should there be – an hierarchy of impact types.
Conclusions and next steps
The HEFCE REF team are compiling notes which will be published at the end of the month or in early March. These will inform Main and Sub Panel discussions about their criteria which will take place over the course of this year. To my mind, panels still face a significant challenge in balancing the sector’s desire for more detailed guidance about how to tackle the presentation of PER based impact with the need not to constrain behaviour by suggesting certain types of PER-based Impact or evidence are more valid than others. What was clear, though, was that impact based on PER was seen as legitimate, valuable and just as capable of being high quality as any other type of impact submitted.