A Retraction Watch email¹ recently highlighted an article by Ben Upton about statistical errors in high impact journals. The criticism is important, because many universities treat impact factors as a de facto correlate for quality when making decisions about faculty hiring and promotion.
Upton described the origin of the data about the errors: “The analysis compared statistical errors from just over 50,000 behavioural and brain sciences articles and the findings of replication studies with journal impact factors and article-level citation counts. It found that articles in journals with higher impact factors tended to have lower-quality statistical evidence to support their claims and that their findings were less likely to be replicated by others.” ¹ Even though these publications are not themselves Information Science journals, the implications matter as an information quality issue whenever impact factors are used to measure performance.
It would be interesting to know exactly which statistical errors were found in the study, if only to be able to warn students against them. The likelihood is that the errors are not simple mathematics, since most scholars today use statistical packages, but rather errors involving poor sampling or errors in understanding the assumptions required for statistical tests.
Regardless of the types of error, the implications go beyond individual cases. Upton warns about “long-standing career inequalities” and cites research by Zachary Horne and Michael R. Dougherty: “Citation counts are known to be lower for women and underrepresented minorities [citations 72–74], and there is some evidence for a negative relationship between impact factor and women authorship [citation 75] and so hiring, tenuring or promoting on their basis may perpetuate structural bias.”² Upton goes on to say: “A European Union-backed agreement on research assessment bars signatories from using impact factors in personnel decisions …”
1: Retraction Watch “RW Daily” email post on 18 August 2022. team@retractionwatch.com
2: Upton, Ben. “Papers in High-Impact Journals ‘Have More Statistical Errors.’” Times Higher Education (THE), August 17, 2022. https://www.timeshighereducation.com/news/papers-high-impact-journals-have-more-statistical-errors.
3: Dougherty, Michael R., and Zachary Horne. “Citation Counts and Journal Impact Factors Do Not Capture Some Indicators of Research Quality in the Behavioural and Brain Sciences.” Royal Society Open Science 9, no. 8: 220334. Accessed August 19, 2022. https://doi.org/10.1098/rsos.220334.
Cover image by 愚木混株 cdd20 on Unsplash
Commenti