The practice of peer review is central to ensuring the quality of research, but a new analysis raises questions about the effectiveness of training programs designed to improve reviewer performance. The analysis, which examined multiple studies, suggests that such training may have little to no impact on the quality of peer reviews.

The research team analyzed 10 studies encompassing 1213 units of analysis, including 722 reviewers and 491 manuscripts. The studies evaluated the impact of various training approaches compared to standard practices.

Key Findings on Peer Reviewer Training

Reviewer training, compared with standard journal practice, may slightly improve peer reviewer ability to detect errors.

The analysis revealed several key findings. For instance, emails reminding peer reviewers to check reporting checklists appeared to have little or no effect on the completeness of reporting, based on evidence from two studies involving 421 manuscripts. Though, one study with 418 reviewers indicated that reviewer training might slightly improve the ability of peer reviewers to detect errors.On average, trained reviewers identified 3.25 out of 9 errors, compared to 2.7 errors identified by untrained reviewers.

Despite this slight advancement in error detection, the analysis found that reviewer training, compared with standard journal practice, may have little to no effect on stakeholders’ assessment of review quality. This conclusion was drawn from six studies involving 616 reviewers and 60 manuscripts.The researchers also expressed uncertainty about the effect of a video course on agreement between reviewers, based on one study with 75 reviewers.Similarly, structured individual feedback on scoring, compared with general data on scoring, may have little to no effect on the change in agreement between reviewers, according to one study with 41 reviewers.

Limitations and Future Research

The researchers cautioned that they have little confidence in most of the evidence due to limitations in the studies, such as a lack of important information and small sample sizes. Additionally, it remains unclear whether the studies measured peer review quality in a valid and reliable way. The evidence is up to date to April 2022.

The analysis underscores the need for larger, well-designed studies to provide more accurate estimates of the effect of training on peer review quality. Such studies should focus on developing and evaluating interventions that can lead to tangible improvements in the rigor and reliability of peer review processes.