January 10, 2023, #25
Many universities and many countries use Journal Impact Factors (JIFs) when assessing the quality of publications for individual scholars, for departments, and for the universities as a whole, and this assessment often has a direct effect on funding. Many administrators like the simplicity of using JIFs as a grading system. An important question is to what degree JIF represents a reliable indicator of quality, and a group of authors decided to investigate. The study was largely British and funding came from “Research England, Scottish Funding Council, Higher Education Funding Council for Wales, and Department for the Economy, Northern Ireland.”¹ The authors warn that: “results are limited by considering a single period (2014-18)“¹ and are “restricted to results from a single country.”¹ The authors relied on “the UK REF”¹ because it “is almost an ideal case in the sense of large-scale expert judgements by people explicitly told [to] ignore the reputation of the publishing journal,”¹ as they note “individual sub-panel members in some UoAs [Units of Assessment] may have disregarded this advice or have been subconsciously influenced, based on their own perceptions of their fields.”¹ The study, which is available on the arXiv preprint server, found that: “[t]he social sciences investigated had weak or moderate correlations”¹, and the social sciences included “library and information science (r=0.528, r=0.267, n=71)”¹. The authors claim that their results “add weight to the evidence that journal impact associates with article quality at least a small amount in all areas of scholarship.”¹ They note, however, that the arts and humanities could be an exception. They also warn that the “correlations are very weak (0.11) to moderate (0.43) for broad fields…. Weaker correlations may reflect non-hierarchical subjects, where journal specialty is more relevant than any journal prestige.”¹ For the iSchools, this is particularly interesting because Information Science methodologically is extremely broad. The results suggest that JIF is not irrelevant as an evaluation tool, but is far from reliable in our field and “confirms that journal impact is not ever an accurate proxy for the quality of individual articles.”¹
1: Thelwall, Mike, Kayvan Kousha, Mahshid Abdoli, Emma Stuart, Meiko Makita, Paul Wilson, and Jonathan Levitt. 2022. ‘In Which Fields Do Higher Impact Journals Publish Higher Quality Articles?’ https://doi.org/10.48550/arXiv.2212.05419.