December 13, 2022, #24
Olavo B. Amaral wrote in Nature on 23 November 2022 that peer review could be fixed by breaking it into stages: “All data should get checked, but not every article needs an expert.”¹ He focuses especially on data quality, and argues that “For most papers, checking whether the data are valid is more important than evaluating whether their claims are warranted…. Undetected errors or fabricated results will permanently damage the scientific record.”¹ He wants to distinguish between basic checks, including the availability of data, and the reliability of the statistical calculations. He raises the question of whether an expert really needs to check every paper, especially since they often do not have the time to do a thorough job, and he recommends using automated tools for some tasks. “In 2015, researchers in the Netherlands developed statcheck, an open-source software package that checks whether P values quoted in psychology articles match test statistics. SciScore — a program that checks biomedical manuscripts for criteria of rigour such as randomization, experiment blinding and cell-line authentication — has screened thousands of COVID-19 preprints.”¹ He noted, however, that such programs do not work if the data are insufficiently standardized, and that merely “checking data cannot guarantee that they were collected as reported, or that they represent an unbiased record of what was observed.”¹ Reproducibility represents a much-discussed topic that garners less academic credit than new articles. Nonetheless, he noted some positive developments: “reproducibility hubs such as the QUEST Center at the Berlin Institute of Health at Charité have been set up to oversee processes across multiple research groups at their institutions.”¹ Recognition is a problem: “These systematic efforts will not become integral to the scientific process unless institutions and funding agencies grant them the status currently enjoyed by journal peer review.”¹ His core concern is that “peer review drains hundreds of millions of hours from researchers but delivers little.”¹ The real problem may not actually be with peer review, but with our unrealistic expectations about what it can accomplish in its current form.
1: Amaral, Olavo B. 2022. ‘To fix peer review, break it into stages’. Nature: World View. 23 November 2022. https://doi.org/10.1038/d41586-022-03791-5.