Written by Jamie Kyte
In today’s fast-paced sports world, data has become a game-changer. Coaches, athletes, and analysts rely on a vast number of statistics, metrics, and insights to gain a competitive edge. A diverse array of performance metrics has been employed both in the sports industry and within research to scrutinize various facets of athletic performance. Notably, a portion of this data is now easily accessible as open-source information, thanks to platforms like FBref (https://fbref.com) and Whoscored (https://www.whoscored.com/)
But here’s the catch: in the era of data-driven decision-making, the quality of the data is paramount. That’s where the concepts of reliability and validity come into play.
“Photo by Fancy Crave – https://unsplash.com/photos/qowyMze7jqg
Imagine you’re a coach preparing your rugby team for a crucial match, a football analyst dissecting a recent game, or a hockey performance scientist fine-tuning training strategies. Your decisions hinge on the data at your disposal, making it imperative that this data is both reliable and valid.
So, what do we actually mean by reliability and validity?
Validity is defined as the capacity of a measurement instrument to accurately reflect the intended attribute it is designed to assess (Atkinson & Nevill, 1998). In the context of performance analysis tools, validity is often evaluated by seeking input from expert coaches within specific sports categories (Torres-Luque et al., 2018).
Reliability in sports performance analysis can be understood as the reproducibility of values derived from a test, assay, or any measurement when repeated trials are conducted on the same individuals. This intra-observer reliability reflects the extent to which measurements consistently yield similar results within the same observer (O’Donoghue, 2009). Additionally, reliability extends to the repeatability of measurements when different observers are involved, known as inter-observer reliability, demonstrating the degree to which measurements remain consistent across various observers (Hopkins, 2000).
It’s worth noting that the reliability of sports notational systems may face certain limitations. Factors such as manual errors, observer inexperience, and the number of observers involved can impact the reliability of the system (Beato et al., 2018). These limitations can lead to unreliable results, potentially misleading coaches and lead to making suboptimal decisions regarding training regimes and match preparations.
While the significance of understanding the reliability of performance analysis tools and measures is widely acknowledged, it is noteworthy that published studies specifically addressing reliability and validity are relatively scarce. A study by Hughes and colleagues sheds light on this issue. They found that out of 72 research papers examined, a staggering 70 percent did not report any reliability analysis of their analysis procedures.
Fortunately, there have been a few studies that delved into the reliability of certain performance analysis products. For instance, Bradley et al. (2007) undertook an assessment of the reliability of ProZone MatchViewer, while Liu et al, 2016 reported on the inter-operator reliability of football match statistics from OPTA Sportsdata. Each of these studies reached a common conclusion: the reliability of the tools in question was contingent on operators having received appropriate training.
While these studies are valuable, providing practitioners with insights into the reliability of specific tools after proper training, it’s essential to recognize that the landscape of performance analysis tools is diverse and continually evolving. Many tools remain unassessed for reliability, leaving a gap in our understanding of their dependability.
To bridge this gap, Dr. John Francis, Michael Bateman from the University of Worcester, Dr. Scott Nicholls from the University of Derby, and Jamie Kyte of the University of Birmingham have embarked on a research project. Their goal? To understand the current practices and perceptions of reliability and validity in sports performance analysis settings.
Through an anonymous online survey, they aim to gather insights from performance analysts around the world. By uncovering the challenges and best practices in the field, they intend to develop a set of ‘best-practice’ recommendations. These recommendations will guide the collection, analysis, and presentation of accurate (reliable) and meaningful (valid) performance analysis data and information to coaches, players, and other stakeholders.
In a world where every second counts, where the finest margins can determine victory or defeat, the reliability and validity of your data are integral to aiding success.
You can contribute to the survey here: https://ucw.onlinesurveys.ac.uk/v_and_r_in_apa
“Photo by https://www.scienceforsport.com/wp-content/uploads/2018/04/Reliability-2-1.jpg”
References:
Atkinson, G. and Nevill, A.M. (1998). Statistical methods for assessing measurement error (reliability) in variables relevant to sports medicine. Sports Medicine, 26(4), pp.217-238.
Beato, M., Coratella, G., and Schena, F. (2018). Correlation between GPS and gyroscope-derived running kinematics in field-based team sports. Sensors, 18(5), p.1507.
Bradley, P.S., O’Donoghue, P., & Wooster, B. (2007). The reliability of ProZone MatchViewer: A video-based performance analysis system. International Journal of Performance Analysis in Sport, 7(3), 17-25.
Gong, B., Wu, Q., Zhang, J., Zhang, C., & Liu, H. (2017). Validity and reliability of data from Champdas Master Match Analysis System in soccer. Journal of Sports Sciences, 35(11), 1034-1042.
Hopkins, W.G., (2000). Measures of reliability in sports medicine and science. Sports Medicine, 30(1), pp.1-15.
Liu, H., Gomez, M. A., Gonçalves, B., Sampaio, J., & Huijing, P. A. (2016). Technical performance and match-to-match variation in elite football teams. Journal of Sports Sciences, 34(6), 509-518.
O’Donoghue, P., (2009). Research methods for sports performance analysis. Routledge.
Torres-Luque, G., Ramirez, A., Cabello-Manrique, D., Nikolaidis, P.T., Alvero-Cruz, J.R., and Ramos-Campo, D.J. (2018). Validity and reliability of the inertial measurement unit for barbell velocity measurement. Journal of Sports Science & Medicine, 17(4), pp.526-533.
J.L.Kyte@bham.ac.uk
Woah! I’m really digging the template/theme of this blog.
It’s simple, yet effective. A lot of times it’s challenging to
get that “perfect balance” between superb usability and visual appeal.
I must say you have done a very good job with this. In addition, the blog loads super quick for me on Opera.
Outstanding Blog!