by Fran Amery, Stephen Bates & Steve McKay
This is the second of two posts on gender and the Research Excellence Framework (you may also be interested in this post on what titles of outputs submitted to the Politics & International Studies Unit of Assessment tell us about (sub-)disciplinary trends).
In our first post, we used the REF submissions data in order to offer a new ‘survey’ of political scientists. We looked at the ratio of men to women across different universities, and with different levels of seniority. In this post, we focus more on the outcomes of the REF and, in particular, the association between the outcomes and the proportion of men and women in each submission.
Sizes of Submissions
The distribution by size of staff is quite skewed: half the institutions submitted to the REF contained one-quarter of the staff, while just the five largest submissions (KCL, Oxford, LSE, Warwick and Birmingham) also contained one-quarter of the staff. As can be seen in Table 1, there is therefore no apparent link between the size of submissions and the proportion of women included. So, for example, looking at this overall scale, it is not the case that women seem to be clustered in smaller submissions. This is important because, on average, smaller submissions tended to achieve lower overall results.
Table 1: Size Distribution
Larger submissions tended to do relatively well, although not all those doing well were large. Figure 1 shows the association between size (and intensity as measured by HEFCE/HESA) and the overall GPA awarded. The size of each point on the graph reflects the size of the submission. The red points are those submitting 84% or more of their eligible staff – dividing the submissions into two groups – with the blue points submitting less than that. We have named the larger and higher achieving submissions in the chart.
Figure 1: Association between Submission Size & Grade Point Average
Unlike the 2008 RAE, it is not possible to link individual academics to particular publications (a bit annoying from our perspective). Instead, at the moment at least, we must rely on a more ‘ecological’ approach by looking at totals within each institution and their REF profiles.
Seniority & Outcomes
As can be seen in Figure 2, having more professors on the staff seemed to be linked to better overall outcomes (again the size of each point on the graph reflects the size of the submission).
Figure 2: Association between Grade Point Average & Professorial Rate
A simple ‘fit line’ is shown, but in reality the fit to the data is relatively weak. The r-squared is 20%, most of the points below the line are the smallest submissions, and the larger submissions mostly stand above the line (i.e. they do ‘better’ than the mere proportion of professors would suggest).
Women & Overall Outcomes
There is a very weak negative correlation between the proportion of women and the outcomes in terms of overall GPA (see Figure 3, r-sq=9%), but again this is not particularly informative as it stands. Once one small submission with a high female proportion is excluded there is no longer a statistically significant association between gender ratio and GPAs.
Figure 3: Association between GPA & Proportion of Women Submitted
Women & Types of Output
Having more women was slightly associated with a smaller proportion of books as outputs (see Figure 4). Even so, different institutions seem to have quite different proportions of books, even for the larger and better-rated submissions.
Figure 4: Association between Women & Proportion of Books
Models of Overall Outcomes
In a multiple linear regression of the overall GPA, higher results were associated with avoiding chapters and edited books as outcomes, having fewer lecturers as a proportion of staff, and having more staff. The gender ratio was not statistically significant in such models.
Observations & Conclusions (with some being More Tentative than Others)
- Submissions with more professors tended to score more highly, ignoring other factors.
- Books were, on average, more likely to be rated as 4* outputs than other kinds of outputs, followed by journal articles (as explicitly stated in the REF Politics panel overview report).
- There was a weak negative association between women and the proportion of books submitted. However, it must be stressed that the emphasis here is on ‘weak’, and data at an individual level (on outputs) would be more informative than the ecological data being used in this post. It is hoped that future work will allow us to focus on the individual level so that we have a better picture of what, if anything, is going on in this regard.
- It is highly plausible that any link between gender and outcomes is being driven by differences in seniority and possibly also by differences in publications profile (i.e. the extent to which men and women are more or less likely to publish monographs, articles, chapters, etc). This is hopefully what our future work will be able to tell us so please watch this space.