Peer-to-peer

Quality and value: Models of quality control for scientific research

Tom Jefferson

A look at alternatives to the peer review system suggests it might be the best bet.

Peer review is used as a quality-control mechanism for biomedical literature (ref. 1). Journals with a formal peer-review system are generally higher profile than those without, and peer review is often a pre-condition for journal indexing in biomedical databases such as PubMed. Variants of the current system or complete alternatives have scarcely been explored, as yet. Two years ago, I and my colleagues carried out an exploratory review of complete and partial alternatives to the current editorial peer-review system, both in and outside biomedical sciences. Here I update and summarise our findings.


The current system has three salient aspects:

1. quality assurance through expert opinions

2. managing competition for publication space

3. the scholarly task of improving scientific knowledge

According to our classification, complete alternatives are those systems that radically depart from all three aspects of editorial peer review. Partial alternatives are those that depart from no more than two aspects of current practice.

We performed a comprehensive layered search, selecting eight methodologically well-defined non-biomedical disciplines (agriculture, astronomy, archaeology, decision-making sciences, law, economics, mathematics, physics). We looked at 31 journals and asked their editors whether they were aware of alternative models of quality control in their disciplines. The answer was disappointing, as they all pursued what seems to be a universal model, whereby every published paper is reviewed, openly or anonymously, by a limited number of ‘expert’ reviewers.

We did not investigate means of access or content dissemination (such as electronic print servers) because we do not consider these as alternative models of peer review, but rather alternative methods of disseminating science whose quality is assured by application of the current model. Perhaps our arbitrary choice of disciplines and sources, coupled with our limited survey may have missed significant alternatives. Readers of Nature may be aware of alternatives, and we would welcome debate (make a comment on this article)

In biomedicine we identified one complete alternative (no peer review) and three partial alternatives: quantitative indicators (to our knowledge not implemented); pre-publication peer review on the Internet (at the Medical Journal of Australia); and the Cochrane collaboration editorial model (at the Cochrane Library). Given the ethical implications of a lack of a quality-control system we did not investigate its abolition further.

Quantitative indicators

We rejected methods based on impact measures, as they concern researchers’ interest in the work of a particular author or authors, rather than the quality of the work itself. Because these measures are of published (or publicly available) work, their use as publication criteria could be disastrous: they would introduce a bias against high-quality work by new and unpublished authors, as well as assuming that future work by an author will be as significant as past work.

Evidence of the effectiveness of indicators as quality-control mechanisms is sparse: we identified only one study (of five major journals) that found a correlation between the number of submissions to a journal and its impact factor; and even then the findings were inconclusive (ref.2).

Pre-publication peer review on the Internet

Open peer review consists of making reviewers’ comments public and/or inviting post-publication comments from readers, which are then published. Such a system was tested at the Medical Journal of Australia, where 60 (81%) authors and 150 (92%) reviewers agreed to take part in a study using website access for pre-publication peer review(ref. 3). Although informal feedback from readers was positive, only 29% of comments led authors to change their manuscripts.

Cochrane collaboration model

The Cochrane editorial model is a collaborative effort to assess the effects of healthcare interventions by publishing and updating systematic reviews in the Cochrane database. This model does not aim to select the best items to publish in a space-limited journal. Editors and authors work together to produce and publish an agreed protocol and then develop it into a review. The editorial process includes pre- and post-publication peer review of protocols and full reviews, a commitment to update the review no more than every two years and a training course for both authors and peer reviewers. The process is completely open; anyone contributing to the review is acknowledged and often editors are also authors and reviewers and vice versa.

The salient characteristic of competition for space is replaced by collaboration, and by reliance on electronic publication, which virtually removes space constraints. We found some unpublished evidence of its effectiveness in an inventory prepared by the Canadian Cochrane Centre that lists examples of the practical impact of Cochrane reviews. By itself, however, this evidence does not seem sufficient to recommend adoption of the system for all biomedical literature.

Maybe there are no meaningful alternatives to the current system, or no one has thought of developing them. Although not perfect, the current model may be the best possible. Should we explore partial alternatives? This would mean that competition would have to give way to collaboration. Such a move would have major implications for science and the publications industry, but we think the goal is worthwhile: improving science.

REFERENCES

1. Jefferson, T., Rudin, M., Brodney, S. & Davidoff, F. Cochrane Database Methodol. Rev.1 , doi:10.1002/14651858.MR000016.pub2 (2006).

2. Joseph, K. S. Br. Med. J. 326, 283 (2003).

3. Bingham, C. M., Higgins, G., Coleman, R. & Van Der Weyden, M. B. Lancet 352, 441–445 (1998).

Tom Jefferson is a medical epedemiologist at the Cochrane Vaccines Field in Rome, Italy. He is an active Cochrane reviewer and one of the editors of the Cochrane Acute Respiratory Infections Group. He co-edited Peer Review in Health Sciences (BMJ, 2000). He acknowledges the help of Marica Ferri, Melanie Rudin, Frank Davidoff and Miranda Cumpston in writing this piece.

See this article in Nature‘s web focus here

Comments

  1. Report this comment

    Adam Drake said:

    Whilst relatively new to the field of research science my observations lead me to the conclusion that the general model of editorial decisions and peer review is good, but not perfect. Whilst exploration of radical new methods may well lead to a paradigm shift such will take time to bed in. The thing that everyone seems to agree on is that a significant amount of time taken by experts to evaluate research is vital to quality control. The problems which seem to occur fall into 3 areas – lack of incentive to do this thankless task, spending too little time on review (so that quality suffers) and finally the controversial issue of prejudice – whether in favour of a respected colleague or aganst a competitor. It seems that a very simple change to the current system would improve all 3 of these problem areas – operate a double blind double open system of peer review.

    When authors submit a paper and it is sent out to review their names and affiliation should not be given to the reviewers. Thus double blind as both parties are ignorant of the others identity. This should prevent at least some discrimination.

    Secondly when a paper is published the Reviewers should be acknowledged as such. Thus double open. This would allow credit to be given to someone who does a good job of reviewing and helps insure that findings are published in an appropriate venue. This should also address the lack of care that can undermine the system as your name is associated with a manuscript and all its findings and their associated experimental flaws. Finally the risk of having your name associated with a manuscript where you have a conflict of interest should discourage the minority of scientists who don’t declare their prior interest.

    I am unaware of any journal that operates such a policy but feel it would be worth evaluating as an improvement to the existing peer review process in the short term – perhaps to be replaced in due course by another system.

Comments are closed.