A Corpus-Based Analysis of REF Output Titles for Politics and International Studies*

Table 1 shows us a ranking of the 100 most popular words used in titles of work submitted to the 2014 REF and 2008 RAE for the Politics & International Studies panel**.  As can be seen and as would probably be expected, the same grammatical words – the, and, of, in and a – occupy the first five places in the same order for both rankings (s is there because of ‘apostrophe-S’; eu is there because the method treats all data as lower case). After that, we can see the kinds of lexical words that, again, we would probably expect to see in a list of popular words with political scientists and IR scholars. So, lots of titles have the words politics, political, policy, international, European, war, theory and power (including this little beauty) in them. These lists are able to tell us a lot about the perennial concerns of the discipline but they aren’t able to tell us very much about growing and declining areas of interest.

Table 1: REF & RAE Politics Word Lists (Sorry for the break in the middle of the table. I couldn’t work out how to insert the whole table at once – the same goes for the other tables below).

Analysing (sub-)disciplinary trends can be achieved through comparing the word lists in Table 1 to produce rankings of keywords for both the 2014 REF and the 2008 RAE. These rankings can be found in Table 2 which lists the top 100 keywords for each. Keywords are words which are unusually frequent, or infrequent, in one collection of words, or corpus, in comparison to another^.

Table 2: REF & RAE Keyword Rankings

So, if we compare the keywords for the 2014 REF with the 2008 RAE, we find that Obama, financial, austerity and crisis are ranked highly. Again, this is what we would probably expect to see given the timing (and the importance) of the global financial crisis, austerity measures and the election of Barack Obama. Yet, beyond these expected keywords, these rankings can help us to identify some interesting – well, I think they’re interesting – possible trends within the discipline:

• Although not appearing very frequently in titles, the high ranking of both experiment and spatial in the 2014 REF list may suggest the growing importance of particular methods and theories to the discipline in the UK. The relatively high ranking of philosophy in the 2008 RAE list may indicate a decline in interest (or that political philosophers were submitted to the Philosophy sub-panel).
• We are no longer as concerned with globalisation/globalization as we were in 2008 or, at least, we now call it something different and/or don’t refer to it in our titles
• The same can be said for technology, industry, things that are communist, and diplomacy, while we’ve become more interested in things to do with the climate, counterinsurgency and counter-terrorism.
• In terms of geography, Afghanistan, Greece, China, Ecuador, Somalia, Rwanda and India have shot up our list of concerns, whereas we’ve become less interested in Germany, Asia (or at least the bits of Asia that are not China, India and Afghanistan), Estonia, Chile, Mozambique and Canada.
• Things European are now not so prominent or important as they were in 2008 (Europa, European, Europe and EMU  all appear in the top 100 keywords for RAE 2008, although European remains very high in the REF 2014 word list ranking).
• We now talk more often of capitalisms than we used to.
• We have turned to Polanyi more often since 2008, presumably to understand the crisis and its aftermath.
• We do not turn to Rawls as often as we used to.
• We are now more interested in the left than in Labour.
• A focus on gender is becoming an increasingly important element of the discipline.
• The presence of und, der and (maybe) die in the 2008 list suggests that we do not publish as much work in German as we used to.
• The presence of Judith and Friedrich but not (presumably) Butler and Nietzsche/Engels suggests we’re becoming politer as a discipline, and/or is simply a reflection that our titles are getting longer (the average length of a title submitted to the 2014 REF and 2008 RAE was 10.7 words and 9.9 words respectively).

Table 3 compares the 2014 REF Politics word list to the 2014 Economics and 2014 Sociology word lists. As would be expected, the keyness figures are much higher in these rankings than when comparing the REF Politics to the RAE Politics and most of the keywords are words that you would expect to find in such a list, relating, as they do, to typical disciplinary concerns (for example, war, security, party, EU, democracy for Politics and IR; market(s), growth, price, wage, inflation for Economics; and life, family, work, urban, class, household for Sociology). There are, however, some more interesting inclusions and omissions, particularly when comparing economics to political science and vice versa:

• Noticeable at the top of the Politics-compared-to-Economic/Sociology and Sociology-compared-to-Politics rankings are politics and the political, and sociology and the social respectively. Noticeable by its absence at the top of the Economics-compared-to-Politics ranking are economics and economic. This appears to be both because political science and sociology focus on economics in a way that economics does not on politics and sociology and because economics does not talk about itself in the same way as the other disciplines do. Thus, for example, while economic and economics are mentioned 83 times and 7 times in political science titles respectively (compared to 56 times and 14 times in economics titles), politics and political are mentioned 5 times and 31 times in economics titles (compared to 406 times and 381 times in political science titles).
• The appearance of crisis and capitalism (and maybe also neo and liberalism) in the top 100 Politics-compared-to-Economics keywords is perhaps surprising. Indeed, capitalism is not mentioned in any economics title at all (although capital is mentioned 55 times).
• The inclusion of words such as evidence, models, equilibrium and games at the top of the Economics-compared-to-Politics list without a corresponding list of theoretical/methodological-type words at the top of the Politics-compared-to-Economics list suggests that political science as a discipline is pluralist and diverse in a way that economics is not.
• Although bioethics is increasingly important for political scientists, sociologists remain much more likely to use words related to biotechnology in their titles (for example, genetic(s), stem, cell, biology).
• The presence of Bourdieu in the Sociology-compared-to-Politics ranking suggests that political scientists still cannot, or do not want to, find a use for him in their work, or at least do not want to shout about using him in their titles (only one person did so).
• In comparison with both economists and sociologists, political scientists like to use the and apostrophes in their titles.
• In comparison with political scientists, economists like to use with in their titles.

Table 3: REF Politics, Economics & Sociology Keyword Rankings

Bis bald.

* This analysis was undertaken by: (1) removing as many duplicate titles as possible (it was not possible to remove all because some titles submitted twice were listed using different punctuation and/or combinations of upper and lower case letters); (2) saving all remaining titles submitted as outputs to REF 2014 and RAE 2008 to plain text files; and then (3) using the wordlist and keyword tools on AntConc, a freeware corpus analysis toolkit for concordancing and text analysis, designed by Prof. Laurence Anthony of  Waseda University Japan.

** The total number of different words, or word types, used in titles submitted to the 2014 REF and the 2008 RAE were 6708 and 6767 respectively. A full list of the word types and keywords are available on request.

^ Using Log Likelihood as the statistical measure, the following significance values apply:

• 95th percentile; 5% level; p < 0.05; critical value = 3.84
• 99th percentile; 1% level; p < 0.01; critical value = 6.63
• 99.9th percentile; 0.1% level; p < 0.001; critical value = 10.83
• 99.99th percentile; 0.01% level; p < 0.0001; critical value = 15.13

See page 7 of here and also here for more.