Science Europe established the Coalition for Advancing Research Assessment (CoARA) in September 2022 and important organisations like the European University Association (EUA) and the European Commission are members of the interim secretariat (2). Some of the principles listed in the document are fairly standard, such as complying with “ethics and integrity rules and practises", and ensuring the "independence and transparency of the data infrastructure and criteria necessary for research assessment". CoARA calls on assessment processes to “respect the variety of scientific disciplines, research types (e.g. basic and frontier research vs. applied research), as well as research career stages”(2). The document goes on to emphasise the primary importance of using qualitative evaluation, including peer review for assessment. Some statements are unusually forceful: “Abandon inappropriate uses in research assessment of journal- and publication based metrics, in particular inappropriate uses of Journal Impact Factor (JIF) and h-index”. CoARA goes on to warn that evaluation should “[a]void the use of rankings of research organisations in research assessment” since “the international rankings most often referred to by research organisations are currently not ‘fair and responsible’, the criteria these rankings use should not trickle down to the evaluation of individual researchers, research teams and research units.” This is unusually strong and clear language for associations more accustomed to avoiding controversy by making recommendations so bland that they will offend no one. Many governments and many institutions prefer to rely on metrics like impact factors because they offer a way to make the assessment process seem more neutral, even more scientific, by using externally generated numerical data. What the document does not explain is that anyone using index data needs to understand how the meaning varies from discipline to discipline and methodology to methodology. A high impact factor may mean that scholars regard the research as important, or just that the topic temporarily generates a high level of interest that will wane over time. Implementing these sorts of recommendations will be hard, and ultimately the document calls on institutions to “[c]ommit resources to reforming research assessment as is needed to achieve the organisational changes committed to”. Committing resources is no guarantee of improvement, but it represents the will to change.
1: https://coara.eu/coalition/coalition/ 2: ‘The Agreement Full Text’. 2022. COARA (blog). Accessed 9 October 2022. https://coara.eu/agreement/the-agreement-full-text/.
Comments