{"id":773,"date":"2015-03-03T11:57:26","date_gmt":"2015-03-03T11:57:26","guid":{"rendered":"http:\/\/polsis.org\/?p=773"},"modified":"2015-03-03T11:57:26","modified_gmt":"2015-03-03T11:57:26","slug":"a-corpus-based-analysis-of-ref-output-titles-for-politics-and-international-studies","status":"publish","type":"post","link":"https:\/\/blog.bham.ac.uk\/polsis\/2015\/03\/03\/a-corpus-based-analysis-of-ref-output-titles-for-politics-and-international-studies\/","title":{"rendered":"A Corpus-Based Analysis of REF Output Titles for Politics and International Studies*"},"content":{"rendered":"<p style=\"text-align:left\"><strong><em>Have you ever wondered what the titles of\u00a0outputs submitted to the REF tell us about disciplinary trends, foci and boundaries? Well, thanks to the wonder of corpus linguistics and\u00a0<a href=\"http:\/\/www.laurenceanthony.net\/\"><span style=\"text-decoration:underline\">Laurence Anthony<\/span><\/a>\u2018s <a href=\"http:\/\/www.laurenceanthony.net\/software.html\"><span style=\"text-decoration:underline\">AntConc<\/span><\/a>,\u00a0wonder no longer.<\/em><\/strong><\/p>\n<div class=\"entry-content\">\n<figure id=\"attachment_713\" aria-describedby=\"caption-attachment-713\" style=\"width: 110px\" class=\"wp-caption alignleft\"><a href=\"http:\/\/blog.bham.ac.uk\/polsis\/wp-content\/uploads\/sites\/42\/2014\/03\/bates-stephen1.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-713 size-full\" src=\"http:\/\/blog.bham.ac.uk\/polsis\/wp-content\/uploads\/sites\/42\/2014\/03\/bates-stephen1.jpg\" alt=\"\" width=\"110\" height=\"146\" \/><\/a><figcaption id=\"caption-attachment-713\" class=\"wp-caption-text\">Dr Stephen Bates is a Lecturer in the Department of Political Science and International Studies at the University of Birmingham<\/figcaption><\/figure>\n<p><strong><em>T<\/em><\/strong><strong><em>able 1<\/em>\u00a0<\/strong>shows us a ranking of the 100 most popular words used in titles of work submitted to the 2014 REF and 2008 RAE\u00a0for the Politics &amp;\u00a0International Studies panel**. \u00a0As can be seen and as would probably be expected, the same <a title=\"Grammatical Words\" href=\"http:\/\/grammar.about.com\/od\/fh\/g\/functionword.htm\" target=\"_blank\"><span style=\"text-decoration:underline\"><span style=\"color:#0066cc\">grammatical words<\/span><\/span><\/a>\u00a0\u2013 <em>the, and, of, in\u00a0<\/em>and\u00a0<em>a \u2013\u00a0<\/em>occupy the first five places in the same order for both rankings (<em>s<\/em> is there because of \u2018apostrophe-S&#8217;; <em>eu <\/em>is there because the method treats all data as lower case). After that, we can see the kinds of <a title=\"Lexical word\" href=\"http:\/\/grammar.about.com\/od\/c\/g\/contentwordterm.htm\" target=\"_blank\"><span style=\"text-decoration:underline\"><span style=\"color:#0066cc\">lexical words<\/span><\/span><\/a> that, again, we would probably expect to see in a list of popular words with political scientists and IR scholars. So, lots of titles have the words <i>politics, political, policy, international, European, war, theory\u00a0<\/i>and\u00a0<em>power\u00a0<\/em>(including this <a title=\"Re-Structuring Power\" href=\"http:\/\/www.palgrave-journals.com\/polity\/journal\/v42\/n3\/abs\/pol20109a.html\" target=\"_blank\"><span style=\"text-decoration:underline\"><span style=\"color:#0066cc\">little beauty<\/span><\/span><\/a>) in them. These lists are able to tell us a lot about the perennial concerns of the discipline but they aren\u2019t able to tell us very much about growing and declining areas of interest.<\/p>\n<p><em><strong>Table 1: REF &amp; RAE Politics Word Lists\u00a0<\/strong><\/em>(Sorry for the break in the middle of the table. I couldn\u2019t work out how to insert the whole table at once \u2013 the same goes for the other tables below).<\/p>\n<p><a href=\"https:\/\/stephenrbates.files.wordpress.com\/2015\/02\/politics-wordlist-11.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-132 size-large\" src=\"https:\/\/stephenrbates.files.wordpress.com\/2015\/02\/politics-wordlist-11.png?w=364&amp;h=1024\" alt=\"Politics Wordlist 1\" width=\"364\" height=\"1024\" \/><\/a> <a href=\"https:\/\/stephenrbates.files.wordpress.com\/2015\/02\/politics-wordlist-21.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-133 size-large\" src=\"https:\/\/stephenrbates.files.wordpress.com\/2015\/02\/politics-wordlist-21.png?w=380&amp;h=1024\" alt=\"Politics Wordlist 2\" width=\"380\" height=\"1024\" \/><\/a><\/p>\n<p>Analysing (sub-)disciplinary trends\u00a0can be achieved through comparing\u00a0the word lists in <strong><em>Table 1\u00a0<\/em><\/strong>to produce rankings of keywords for both the 2014 REF and the 2008 RAE. These rankings\u00a0can be found in <strong><em>Table 2 <\/em><\/strong>which lists the top 100 keywords for each. Keywords are\u00a0words which are\u00a0unusually frequent, or infrequent, in one collection of words, or corpus, in comparison to another^.<\/p>\n<p><em><strong>Table 2: REF &amp; RAE Keyword Rankings<\/strong><\/em><\/p>\n<p><a href=\"https:\/\/stephenrbates.files.wordpress.com\/2015\/02\/politics-keyword-1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-130 size-large\" src=\"https:\/\/stephenrbates.files.wordpress.com\/2015\/02\/politics-keyword-1.png?w=501&amp;h=1024\" alt=\"Politics Keyword 1\" width=\"501\" height=\"1024\" \/><\/a> <a href=\"https:\/\/stephenrbates.files.wordpress.com\/2015\/02\/politics-keyword-2.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-131 size-large\" src=\"https:\/\/stephenrbates.files.wordpress.com\/2015\/02\/politics-keyword-2.png?w=522&amp;h=1024\" alt=\"Politics Keyword 2\" width=\"522\" height=\"1024\" \/><\/a><\/p>\n<p>So, if we compare the keywords for the 2014 REF with the 2008 RAE, we find that <em>Obama, financial, austerity\u00a0<\/em>and\u00a0<em>crisis\u00a0<\/em>are ranked highly. Again, this is what we would probably expect to see given the timing (and the importance) of the global financial crisis, austerity measures and the election of Barack Obama.\u00a0Yet, beyond these expected\u00a0keywords, these rankings can help us to identify some interesting \u2013 well, I think they\u2019re interesting \u2013 possible trends within the discipline:<\/p>\n<ul>\n<li>Although not appearing very frequently in titles, the high ranking of both\u00a0<em>experiment\u00a0<\/em>and\u00a0<em>spatial <\/em>in the 2014 REF list may\u00a0suggest the growing importance of particular methods and theories to the discipline in the UK. The relatively high ranking of philosophy in the 2008 RAE list may indicate a decline in interest (or that political philosophers were submitted to the Philosophy sub-panel).<\/li>\n<li>We are no longer as concerned with <em>globalisation\/globalization<\/em> as we were in\u00a02008 or, at least, we now call it something different and\/or don\u2019t refer to it in our titles<\/li>\n<li>The same can be said for <em>technology<\/em>, <em>industry, <\/em>things that are\u00a0<em>communist,<\/em>\u00a0and <em>diplomacy<\/em>, while we\u2019ve become more interested in things to do with the <em>climate<\/em>, <em>counterinsurgency<\/em> and <em>counter-terrorism<\/em>.<\/li>\n<li>In terms of geography, <em>Afghanistan<\/em>, <em>Greece<\/em>,\u00a0<em>China, Ecuador, Somalia,\u00a0<\/em><em>Rwanda<\/em>\u00a0and <em>India\u00a0<\/em>have shot up our list of concerns, whereas we\u2019ve become less interested in <em>Germany<\/em>,\u00a0<em>Asia<\/em> (or at least the bits of Asia that are not China, India and Afghanistan), <em>Estonia<\/em>, <em>Chile,<\/em> <em>Mozambique\u00a0<\/em>and\u00a0<em>Canada<\/em>.<\/li>\n<li>Things European are now not so prominent or important\u00a0as they were in 2008 (<em>Europa, European, Europe<\/em>\u00a0and <em>EMU\u00a0\u00a0<\/em>all appear in the top 100 keywords for RAE 2008, although <em>European <\/em>remains very high in the REF 2014 word list ranking).<\/li>\n<li>We now talk more often of\u00a0<em>capitalisms <\/em>than we used to.<\/li>\n<li>We have turned\u00a0to <em>Polanyi<\/em> more often since 2008, presumably to understand the crisis and its aftermath.<\/li>\n<li>We do not turn to\u00a0<em>Rawls\u00a0<\/em>as often as we used to.<\/li>\n<li>We are now more interested in\u00a0the\u00a0<em>left\u00a0<\/em>than in\u00a0<em>Labour.<\/em><\/li>\n<li>A focus on <em>gender<\/em> is becoming an increasingly important element of the discipline.<\/li>\n<li>The presence of <i>und<\/i>, <em>der<\/em>\u00a0and (maybe)\u00a0<em>die\u00a0<\/em>in the 2008 list suggests that we do not\u00a0publish as much work in German as we used to.<\/li>\n<li>The presence of\u00a0<em>Judith\u00a0<\/em>and\u00a0<em>Friedrich\u00a0<\/em>but not (presumably)<em>\u00a0<\/em>Butler and Nietzsche\/Engels suggests we\u2019re becoming politer as a discipline, and\/or is simply a reflection that our titles are getting longer (the average length of a title submitted to the 2014 REF and 2008 RAE was 10.7 words and 9.9 words respectively).<\/li>\n<\/ul>\n<p><strong><em>Table 3\u00a0<\/em><\/strong>compares the 2014 REF Politics word list to the 2014 Economics and 2014 Sociology word lists. As would be expected, the keyness figures are much higher in these rankings than when comparing the REF Politics to the RAE Politics\u00a0and\u00a0most of the keywords are words that you would expect to find in such a list, relating, as they do, to typical disciplinary concerns (for example,\u00a0<em>war, security, party, EU, democracy\u00a0<\/em>for Politics and IR;\u00a0<em>market(s), growth, price, wage, inflation\u00a0<\/em>for Economics; and\u00a0<em>life, family, work, urban, class, household\u00a0<\/em>for Sociology). There are, however, some more interesting inclusions and omissions, particularly when comparing economics to political science and <em>vice versa<\/em>:<\/p>\n<ul>\n<li>Noticeable at the top of the Politics-compared-to-Economic\/Sociology and Sociology-compared-to-Politics rankings are <em>politics\u00a0<\/em>and the\u00a0<em>political<\/em>, and\u00a0<em>sociology\u00a0<\/em>and the\u00a0<em>social\u00a0<\/em>respectively. Noticeable by its absence at the top of the Economics-compared-to-Politics ranking are\u00a0<em>economics<\/em>\u00a0and <em>economic<\/em>.<em>\u00a0<\/em>This appears to be both because political science and sociology focus on\u00a0economics in a way that economics does not on politics and sociology and because economics does not talk about itself in the same way as the other disciplines\u00a0do. Thus, for example<em>, <\/em>while<em>\u00a0<\/em><em>economic<\/em> and <em>economics <\/em>are\u00a0mentioned 83 times and 7 times in political science titles respectively (compared to\u00a056 times and\u00a014 times in economics titles),\u00a0<em>politics\u00a0<\/em>and\u00a0<em>political\u00a0<\/em>are mentioned 5 times and 31 times in economics titles (compared to 406 times and 381 times in political science titles).<\/li>\n<li>The appearance of\u00a0<em>crisis\u00a0<\/em>and\u00a0<em>capitalism<\/em>\u00a0(and maybe\u00a0also\u00a0<em>neo\u00a0<\/em>and\u00a0<em>liberalism<\/em>)<em>\u00a0<\/em>in the top 100 Politics-compared-to-Economics keywords is perhaps surprising. Indeed, <em>capitalism<\/em> is not mentioned in any economics title at all (although\u00a0<em>capital\u00a0<\/em>is mentioned 55 times).<\/li>\n<li>The inclusion of words such as\u00a0<em>evidence, models, equilibrium<\/em>\u00a0and <em>games<\/em> at the top of the Economics-compared-to-Politics list without a corresponding list of theoretical\/methodological-type words at the top of the Politics-compared-to-Economics list suggests that\u00a0political science as a discipline is\u00a0pluralist and diverse in a way that economics is not.<\/li>\n<li>Although bioethics is increasingly important\u00a0for political scientists, sociologists remain much more likely to use words related to biotechnology in their titles (for example, <em>genetic(s),\u00a0stem, cell, biology<\/em>).<\/li>\n<li>The presence of\u00a0<em>Bourdieu\u00a0<\/em>in the Sociology-compared-to-Politics ranking suggests that political scientists still cannot, or do not want to, find a use for\u00a0him in their work, or at least do not want to shout about using him in their titles (only one person did so).<\/li>\n<li>In comparison with both economists and sociologists, political scientists like to use\u00a0<em>the <\/em>and apostrophes in their titles.<\/li>\n<li>In comparison with political scientists, economists like to use\u00a0<em>with\u00a0<\/em>in their titles.<\/li>\n<\/ul>\n<p><b><i>Table 3: REF Politics, Economics &amp;\u00a0Sociology Keyword Rankings<\/i><\/b><\/p>\n<p><a href=\"https:\/\/stephenrbates.files.wordpress.com\/2015\/02\/pol-econ-soc-keyword-1.png\"><img decoding=\"async\" class=\"alignnone wp-image-134 size-large\" src=\"https:\/\/stephenrbates.files.wordpress.com\/2015\/02\/pol-econ-soc-keyword-1.png?w=960&amp;h=819\" alt=\"Pol Econ Soc Keyword 1\" \/><\/a><a href=\"https:\/\/stephenrbates.files.wordpress.com\/2015\/02\/pol-econ-soc-keyword-2.png\"><img decoding=\"async\" class=\"alignnone wp-image-135 size-large\" src=\"https:\/\/stephenrbates.files.wordpress.com\/2015\/02\/pol-econ-soc-keyword-2.png?w=960&amp;h=826\" alt=\"Pol Econ Soc Keyword 2\" \/><\/a><a href=\"https:\/\/stephenrbates.files.wordpress.com\/2015\/02\/pol-econ-soc-keyword-3.png\"><img decoding=\"async\" class=\"alignnone wp-image-136 size-large\" src=\"https:\/\/stephenrbates.files.wordpress.com\/2015\/02\/pol-econ-soc-keyword-3.png?w=960&amp;h=544\" alt=\"Pol Econ Soc Keyword 3\" \/><\/a><\/p>\n<p>Bis bald.<\/p>\n<hr \/>\n<p>* This analysis was undertaken by: (1) removing as many duplicate titles as possible (it was not possible to remove all because some titles submitted twice were listed\u00a0using different punctuation and\/or combinations of upper and lower case letters); (2) saving all remaining titles submitted as outputs to REF 2014 and RAE 2008 to plain text files; and then (3) using the wordlist and keyword tools on <em>AntConc,\u00a0<\/em>a freeware corpus analysis toolkit for concordancing and text analysis, designed by Prof. <a title=\"Laurence Anthony\" href=\"http:\/\/www.laurenceanthony.net\/\" target=\"_blank\"><span style=\"text-decoration:underline\"><span style=\"color:#0066cc\">Laurence Anthony<\/span><\/span><\/a>\u00a0of \u00a0Waseda University Japan<em>.<\/em><\/p>\n<p>** The total number of different words, or word types, used in titles submitted to the 2014 REF and the 2008 RAE were 6708 and 6767 respectively.\u00a0A full list of the word types\u00a0and keywords\u00a0are available on request.<\/p>\n<p>^ Using Log Likelihood as the statistical measure, the following significance values apply:<\/p>\n<ul>\n<li>95th percentile; 5% level; p &lt; 0.05; critical value = 3.84<\/li>\n<li>99th percentile; 1% level; p &lt; 0.01; critical value = 6.63<\/li>\n<li>99.9th percentile; 0.1% level; p &lt; 0.001; critical value = 10.83<\/li>\n<li>99.99th percentile; 0.01% level; p &lt; 0.0001; critical value = 15.13<\/li>\n<\/ul>\n<p>See page 7 of\u00a0<a title=\"Antconc help\" href=\"http:\/\/www.laurenceanthony.net\/software\/antconc\/releases\/AntConc343\/help.pdf\" target=\"_blank\"><span style=\"text-decoration:underline\"><span style=\"color:#0066cc\">here<\/span><\/span><\/a> and also <a title=\"Ucrel LL\" href=\"http:\/\/ucrel.lancs.ac.uk\/llwizard.html\" target=\"_blank\"><span style=\"text-decoration:underline\"><span style=\"color:#0066cc\">here <\/span><\/span><\/a>for more.<\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>Have you ever wondered what the titles of\u00a0outputs submitted to the REF tell us about disciplinary trends, foci and boundaries? Well, thanks to the wonder of corpus linguistics and\u00a0Laurence Anthony\u2018s AntConc,\u00a0wonder no longer. Table 1\u00a0shows us a ranking of the 100 most popular words used in titles of work submitted to the 2014 REF and &hellip; <a href=\"https:\/\/blog.bham.ac.uk\/polsis\/2015\/03\/03\/a-corpus-based-analysis-of-ref-output-titles-for-politics-and-international-studies\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;A Corpus-Based Analysis of REF Output Titles for Politics and International Studies*&#8221;<\/span><\/a><\/p>\n","protected":false},"author":144,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[11],"tags":[],"class_list":["post-773","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/blog.bham.ac.uk\/polsis\/wp-json\/wp\/v2\/posts\/773","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.bham.ac.uk\/polsis\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.bham.ac.uk\/polsis\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.bham.ac.uk\/polsis\/wp-json\/wp\/v2\/users\/144"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.bham.ac.uk\/polsis\/wp-json\/wp\/v2\/comments?post=773"}],"version-history":[{"count":0,"href":"https:\/\/blog.bham.ac.uk\/polsis\/wp-json\/wp\/v2\/posts\/773\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.bham.ac.uk\/polsis\/wp-json\/wp\/v2\/media?parent=773"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.bham.ac.uk\/polsis\/wp-json\/wp\/v2\/categories?post=773"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.bham.ac.uk\/polsis\/wp-json\/wp\/v2\/tags?post=773"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}