The Five(ish) Ws of Impact Evaluation

Published: Posted on

Guest blog post by Beth Clewes Impact Development Officer, College of Social Sciences

With REF2021 submission creeping ever closer (less than 2 years to go!), and the impact agenda becoming further embedded into research activities in universities across the UK, a question that is increasingly being considered by academics and professional services staff alike is ‘how can we effectively measure the impacts of research?’.

While this is, in many ways, an impossible question which will elicit the inevitable answer of ‘it depends on the research project and your impact aims’, fear not! Below is some practical advice for how you can approach planning your impact evaluation activities.

These questions are intended to be applicable to impact evaluation at any stage of the research process. While increasingly the emphasis is on building impact evaluation into the initial project design, it’s still possible to build a strong case for impact from evidence collected at a later stage.

‘The Why’

First of all, it’s important to think about why you are undertaking an impact evaluation. Is it because you’re preparing for REF2021 or future REFs? Is it a requirement of your funder that you can demonstrate the impacts of your research? Are you interested in measuring the impacts of your current project to inform your future research activities?

The motivations behind why you’re evaluating impact will determine how you want to measure changes, as well as other logistics such as how extensive your evaluation methodology will be. Evidence collection for REF may need to be more rigorous than if you’re undertaking an evaluation to inform other projects you’re considering, as the links between your research and the changes you’re claiming will need to be more robust for an impact case study.

‘The What’

The next step in this process is to think about what change you are trying to measure. It’s likely that you already have an idea of what change(s) you’re anticipating within your intended beneficiary groups, and this will be linked to the end goal of why you are undertaking the research in the first place.

If you find that the end goal you’ve envisioned is quite broad, or that it’s actually a longer-term impact objective that is unlikely to come to fruition in a single research project, it may be helpful to think of more realistic, smaller-scale changes that could emerge from your current activities.

‘The Where’

Where can you focus your evaluation activities to best measure the impacts you’re claiming? Focus your evaluation efforts on the areas of change that you believe are the strongest based on your engagement and dissemination activities.

In a similar way to designing a research project, where your choice of methods and participants needs to be appropriate to the research question(s), your approach to impact evaluation should be tailored to the changes you’re aiming to demonstrate. For example, your evaluation activities could be determined by what is most fitting for the ‘type’ of impact you’re claiming, or what methods work best for a certain beneficiary or stakeholder group.

‘The Who’

Next up is who has been affected by the changes brought about by your research? Who are the key stakeholders or beneficiaries you’re intending to reach, and who do you believe will experience the most tangible change(s) from your research and engagement activities? These will be the people you need to approach during the evaluation. It can be useful to list these people or groups and then think about the most effective way of approaching them to best evidence the changes they’ve encountered.

Another ‘who’ question to consider is who do you think is best placed to answer questions about the impacts you’re claiming, and your role in it? For some, the answer may be the same as for the previous question. For others, such as those claiming policy impacts, those most affected by changes may not be in the best position to corroborate the role the research has had in it. In cases like these, it may be more appropriate to approach key players involved in enacting the changes to evidence how your work influenced the decision-making process. When thinking in REF terms, it is especially important that those partaking in your evaluation activities are able to establish that the impacts you’re claiming are directly linked to your research.

As well as considering the relevance of those you’re approaching in your evaluation activities, there are also practical considerations in terms of who you feasibly have access to. If you’re currently at the planning stages of a research project, it may be worth building in a method of capturing permission from individuals if you’re considering following up with them for evaluative purposes. If you’re further on in the research process and still have connections with your key stakeholders or beneficiaries, it might be useful to check in now to see if you can contact them as part of your evaluation, assuming you don’t think it would be damaging to your relationship with them to do so.

‘The How’

The last step in this process is to think about how you can evaluate any secondary impacts that may emerge. As you don’t have to be directly involved in secondary impacts to claim them as your own, you may have to implement different evaluation methods to those used to capture your ‘direct’ impacts. For example, your ‘primary’ impact on policy may have led to secondary changes in social welfare or the practice of certain organisations, but the evaluative activity you use for one may not be appropriate to demonstrate the other. Anticipating your secondary impacts during the planning process will mean that you can design your evaluation to account for all of the changes you’re attempting to measure. Comparative and longitudinal data can both be useful in order to show the state of play before and after your intervention.

While on the surface the measurement of secondary impact seems more important for strengthening REF impact case studies, it can also be useful to know what secondary changes have occurred if you’re measuring impact for impact’s sake. The knowledge of secondary groups who have benefitted from your research could give you an idea of new stakeholders or beneficiaries to collaborate with on future projects.

Note: If your motivations for evaluating impact are REF-related, you may also want to think about ‘The When’. When did the impacts you’re evaluating take place? For REF2021, the impacts you’re claiming need to have occurred between 1st August 2013 and 31st July 2020, so if you’re submitting an impact case study it’s worth keeping this timeframe in mind during the evaluation process.

To discuss how you can demonstrate or extend the reach and significance of your impacts and to find out what resources and support are available to you, get in touch with your Impact Development Officer or the Research Planning Partner for your School or College.

Leave a Reply

Your email address will not be published. Required fields are marked *