Reducing the peer-reviewer’s burden

NChemBioMay.gif

Nature Chemical Biology ( 6, 307; 2010) asks in its May Editorial: what can be done to reduce the burden on scientific referees while ensuring the continuity and quality of peer review?

Peer review, in which scientists critically evaluate ideas, results and scientific models, is central to all scientific discourse and especially critical for ensuring the quality of the scientific literature. Researchers profit from the peer review process in their roles as authors, where it improves their published papers. They also benefit as referees by getting a broad view of leading studies in their field and by enhancing the rigor of their discipline’s published literature. Yet, in the current climate of greater scientific output, peer review is just one component of the expanding workload of scientists. Last month (Nat. Chem. Biol. 6, 245, 2010) we answered some frequently asked questions about the peer review process at Nature Chemical Biology. This month we suggest ways to broaden the voices participating in peer review and reduce its demands on individual scientists.

As technological advances have equipped scientists with new tools to probe scientific questions in unprecedented ways, the pace of research has expanded significantly, particularly in interdisciplinary areas. This more competitive landscape has placed increasing pressure on scientists to publish their research in leading journals. More manuscripts are being written and a higher burden for each manuscript to include more and better data puts corresponding pressure on the peer review system. Unfortunately, this is happening in an environment where the ‘to do’ lists of scientists are already becoming unmanageable.

Despite everyone’s best efforts, the slowest step in the publication process remains the evaluation of manuscripts by anonymous experts. All journals, including Nature Chemical Biology, strive to balance the desires of authors for expeditious review with our need for the high quality referee feedback necessary for making informed editorial decisions. Given these competing demands, the scientific community needs to find ways to reduce the burden of peer review, while making sure that it fulfills its central role in the advancement of science.

Journals can do their part by guaranteeing that peer review responsibilities are well distributed across their communities. At Nature Chemical Biology, we limit how frequently individual scientists are contacted to serve as referees, which helps to manage their reviewing load and balance their scientific influence at the journal.

Nature Chemical Biology is committed to expanding our referee database to ensure that our referee pool reflects the scientific, demographic and geographical diversity of our authors and readers. In adding scientists to our database of potential reviewers, the editors rely on our familiarity with the different fields of chemical biology, as well as contacts made at conferences or lab visits. Authors consistently provide excellent referee suggestions, and scientists who decline referee requests frequently suggest appropriate alternatives. We also use online resources and search engines to identify potential referees, but this mechanism requires that scientists be ‘findable’ on the Internet. Thus, we encourage scientists, universities and companies to consider this as they design and update their websites.

Developing better systems to train young scientists in the peer review process is another main component of this strategy. We urge principal investigators to work with their colleagues and institutions to establish formal peer review training in their curricula, focusing on the intellectual aspects of review, such as how to assess the aims and technical merit of scientific studies. However, they also need to include training on the practical matters of how to express constructive criticism clearly in writing and should examine the professional and ethical dimensions of peer review. Such approaches will help young scientists develop peer reviewing skills, which will also shape students’ views of how to design and evaluate their own scientific work.

After review, we routinely send anonymous copies of all referee reports to the referees of each manuscript; we encourage all referees to examine these and discuss them with any collaborating co-referees, as the combined reports provide a useful way to view the review process more comprehensively and learn from the comments of other scientists.

Taken together, the contributions of individual referees as well as training programmes in peer review will expand the peer review network to incorporate the broadest base of referees necessary for supporting the expanding scientific community and literature.

What Nature Chemical Biology asks of its reviewers

nchembioapr.gif

Excerpted from Nature Chemical Biology 6, 245 (2010).

Peer review remains the primary mechanism for maintaining high standards and ensuring the completeness and accuracy of scientific studies. It also provides practical feedback to authors, which leads to better papers in the scientific literature. Over the years, the editorial team and Nature Chemical Biology authors have been fortunate to work with a diverse and conscientious group of referees. These scientists have consistently provided us with timely and thoughtful feedback on the novelty, technical merit and potential significance of manuscripts. Here, we discuss some specific aspects of how the peer review process works at Nature Chemical Biology and outline our expectations of our referees.

In agreeing to provide comments, reviewers should be aware that, even though they are volunteering their efforts, they are making a number of important commitments to the authors and the journal. First, we ask that referees declare any potential conflicts (positive or negative) or concerns about expertise before agreeing to review a manuscript. Second, by accepting a review request, referees must treat the manuscript as confidential and not distribute or disclose its contents before it is published. Third, reviewers agree to provide their comments by our deadline—typically two weeks for new submissions. To ensure that the review process stays on track, we send reminders to referees as their deadline approaches and follow up on late reports. Fourth, scientists should be aware that by accepting a review request they need to make themselves available to look at revised versions of the manuscript. Revised manuscripts almost always include new experiments, and we rely on referees to assess whether these new data are technically sound and address the earlier concerns. The editorial team realizes that this requires more effort, so we do our best to ensure that the authors have made a serious effort to address the referee concerns before sending a paper back to review. We feel that these referee commitments are essential for effective peer review; as a result, we would prefer that scientists decline a review request and suggest other potential experts if they feel unable to fulfill these obligations.

At Nature Chemical Biology, we look for several things in referee comments, which are detailed more fully at our Authors & Referees site. The most useful reports outline the arguments for or against publication and provide the authors with specific suggestions of experiments and revisions necessary to strengthen the results and conclusions of the study. In their comments, referees should provide a summary of the major claims of the paper, a detailed assessment of its technical merit, an evaluation of how the paper fits into the current literature, and opinions on the study’s potential significance for the discipline and general appeal to chemical biologists. In assessing experimental work, referees should consider whether the reported experimental details are scientifically and statistically sound, support the main conclusions and are described in sufficient detail to enable reproduction of the results.

Though we rely heavily on the advice of referees, the editorial team makes the final decisions on which papers will be published in Nature Chemical Biology. We take this responsibility quite seriously, in the earnest belief that these decisions will be best for our readers and for the field. Providing a thoughtful scientific review requires effort and time. We are extremely grateful for our referees’ diligence and ongoing commitment to supporting the high standards of the journal.

Nature Medicine peers into review

NMedMar.gif

In its March Editorial, Peering into review, Nature Medicine addresses how the peer review process can be frustrating to researchers eager to get their work published. Changes to the process might be warranted, says the journal —but only if they are based in fact, not conjecture.

The Editorial discusses a recent “open letter” written by a group of stem-cell researchers about what they see as obstructive and unreasonable reviews delaying publication of their research. From the Editorial:

Publication of referees’ comments in full may affect the quality of the reviews, leading to more cautious and restrained comments. It is difficult to ascertain how much the quality of reviews would be compromised by adopting these measures; however, previous attempts with open peer review suggest that referees are less likely to provide a direct and detailed evaluation of the report. Authors may also be reluctant to adopt this strategy, as publication of earlier reviews may expose flaws that were addressed in later submissions (or never addressed satisfactorily) but nonetheless color interpretation of the findings. Finally, most articles undergo numerous rounds of review. As such, it is difficult to evaluate the importance of comments made in the first round of review without reference to that earlier version of the manuscript. Simply publishing the reviewers’ comments as a whole removes them from the context in which the decisions were made and may place undue emphasis on specific points of concern.

We do not want our readership to feel that the editors are indifferent to the concerns voiced by the stem cell researchers. The peer review process might be improved, but it remains unclear whether publishing reviewers’ comments would provide a measurable advantage that outweighs the drawbacks of adopting this approach. When properly managed, confidential peer review allows for a fair and unbiased assessment and ensures that we publish papers of the highest quality.

Comments from readers are invited at Spoonful of Medicine, the journal’s blog.

Nature Genetics wants proportional representation

NGenMar.gif

This is a shortened version of the Editorial in the March issue of Nature Genetics (42, 187; 2010).

Nature Genetics publishes papers from a very broad geographical catchment, and we invite peer referees from among the world’s best genetics researchers in order to attract and publish papers of a uniformly high standard. We need to do more to recruit outstanding referees from under-represented regions.

Editors make lab visits and meet authors at conferences not only to learn about research that is ready to be published in the journal, but also to hear critical comments from authors about research published in the journal. These conversations are often a good way to identify potential new peer referees who will be committed to raising the standard of the papers we publish. Because we are full-time editors and we do not have a formal editorial board of advisors, constant feedback about our own decisions and the advice we receive from referees helps shape our ever-improving criteria for accepting manuscripts. New contacts also allow us move the focus and expand the scope of the journal to take advantage of fruitful fields of research as they open up.

We are proud that genetics and genomics are among the most global fields of science, bringing us diverse projects from many countries as well as international collaborative research—more than half of the research articles we publish are the products of more than one country. We are also concerned about understanding disparities by country and region in the ratio of submissions to successful publications. Yes, we decide editorially which submissions to send to review (currently about 20%) and when the papers are ready to publish, but the advice the authors and we receive during review is key. So is quality. For us simply to send more papers from less successful regions out for review is not a sustainable way to achieve more publications.

Although we need to work with “calibrated” expert referees who can consistently provide us with the best advice, we are constantly adding referees to the papers we send to review. This process ensures that new referees see more excellent papers, understand the standards applied by referees and editors, and make their own advice clearer and more useful. We believe that researchers who act as referees also benefit because experience gained while reviewing other people’s papers will make them more likely to have success with their own papers here and at all high-quality journals. Regional discrepancies will be progressively corrected as we recruit referees. Most importantly for us, this journal will incorporate their advice and standards and continue to grow.

Nature Genetics journal homepage.

Nature Genetics guide to authors.

What Nature Physics wants

Peer review is the cornerstone of scientific publishing. But it isn’t always clear exactly what Nature Physics expects of its referees. The journal explains in its November Editorial (5, 775; 2009). “Whatever you think about a paper, it is vital to explain to us exactly why you think it. Your colleagues among the other reviewers may disagree with your assessment, and we do not base our decisions on a show of hands. Hence detailed critiques carry more weight in informing our decisions than terse affirmations one way or the other (in most cases we would disregard the latter, regardless of who supplied it). A further point to consider is whether the work presented in a paper is similar to what has been done before — in such a case, please explain exactly what has been done previously and indicate where it was published.” The Editorial outlines the journal’s peer-review process, what the editors look for in a review, how to write the review, and how the editors make their decisions. “Peer review is essential for maintaining the integrity of the scientific record. It’s well worth the effort. And we thank all of you who make it.”

Nature Chemistry on improving peer review

Perceived lapses in the peer-review process often receive a lot of attention, but the majority of researchers declare themselves satisfied with the system even though they would like to improve it. If it is imperfect or broken, how do we fix it? This question is addressed in the November Editorial of Nature Chemistry ( 1, 585; 2009), in light of some blog commentaries which identified prior publications that had not been referenced in a journal paper.

Open peer-review experiments have generally not been very successful because reviewers are less likely to make forthright comments in an open forum. Double-blind peer review is another option, but one must consider the role of the editor who oversees the process, as well as the difficulties of effectively hiding the identity of authors in smaller fields from other experts — especially when many authors regularly cite and discuss their previous work. The Editorial concludes:

“The Royal Society of Chemistry’s Dalton and Faraday discussion meetings provide a unique mix of traditional peer review coupled with both comment (by peers) and responses from the authors, but require members of a particular research community to assemble at a conference. It is in some ways similar to the grant proposal review process at, for example, the US ”http://grants.nih.gov/grants/peer_review_process.htm">National Institutes of Health. However, such a process is clearly not a viable option for every one of the vast number of papers submitted for publication. The journal Atmospheric Chemistry and Physics uses a system in which, after initial assessment by an associate editor, manuscripts are posted online for comment. After referee reports are received, these are also posted online with the manuscript along with author rebuttals. If eventually accepted, a paper is formally published in the journal, whereas those that are not remain available (and citable) as online ‘discussions’. This differs from the preprint servers Nature Precedings and arXiv because there is an initial assessment of the suitability of the work (based on more than just scope).

Perhaps a hybrid system could be the solution. Traditional peer review, and a decision to publish, could be followed by a fixed period in which any interested party could post questions or comments and the authors are given the opportunity to respond — all moderated by an editor — before a final version of the article (including comments and responses) is preserved for the record. This would again require a large change in the habits of the community — authors, reviewers and publishers — and previous experiments with commenting on published papers have been far from conclusive."

Sense about Science peer-review survey 2009.

Nature journals’ peer review policy and Editorials on the subject.

Myles Allen on planetary boundaries and peer review

This post is by Myles Allen of the University of Oxford:

As a vocal supporter of the traditional system of scientists communicating through peer-reviewed channels ( Nat. Geosci. 1, 209; 2008 and associated debate at Peer-to-Peer), I was hesitant about writing a critical Commentary on the Feature ‘A safe operating space for humanity’ by Johan Rockström et al in the 24 September issue of Nature (Nature 461, 472-475; 2009) in a non-peer-reviewed forum. The Nature and Nature Reports Climate Change editors had clearly thought through this argument: the Feature was not itself peer-reviewed, so no golden rules would be broken in publishing a series of commentaries alongside it in Nature Reports Climate Change.

The problem is that packing the point into a few hundred words, and slipping into the usual bloggers’ trap of feeling you have to shout loudly on the internet or no-one will listen, means that the end result reads very black-and-white. Corresponding with Johan Rockström over the past week, it emerges we agree on far more than the tone of my Commentary, ‘Tangible targets are critical’ (Nat. Rep. Climate Change doi:10.1038/climate.2009.95), probably implies. I understand that Rockström et al. had not originally intended to make the link between a six-degree climate sensitivity, the two degrees target and 350 p.p.m. a focal point (as I read it) of their Feature. Likewise, they observe, reasonably enough, that limiting cumulative carbon dioxide emissions to one trillion tonnes of carbon is just another way of framing the climate boundary, with (as I acknowledge) remarkably similar implications to 350 p.p.m.

If this had been done the old-fashioned way, Rockström et al. would almost certainly have had to qualify their reliance on a six-degree climate sensitivity in the course of the usual to-and-fro with referees. And I in turn would have toned down a lot of my objections. The end result would undoubtedly have been blander on both sides, but would that really have been much of a loss? Following what was, for me, an experiment, I still feel it is very much an open question whether scientific communication in general benefits from direct publication rather than allowing rough edges to be smoothed off through traditional peer-review.

Myles Allen is at the University of Oxford, Parks Road, Oxford, OX1 3PU, UK.

-—————————————

Context:

Johan Rockström and his co-authors argue in a Nature Feature that to avoid catastrophic environmental change, humanity must stay within defined ‘planetary boundaries’ for a range of essential Earth-system processes. If one boundary is transgressed, then safe levels for other processes could also be under serious risk, they caution. Seven expert commentaries respond to this proposal in Nature Reports Climate Change, one of which is by Myles Allen. All these articles can be accessed from this index page.

There is an associated Nature podcast in which Johan Rockström is interviewed and editor Ehsan Masood provides further analysis.

Further reading:

A longer paper upon which the Nature Feature is based, ‘Tipping towards the unknown’, is available at the Stockholm Resilience Centre.

Nature news special: The road to Copenhagen.

NSMB speaks up for peer-reviewers

Manuscript peer reviewing is at the heart of the scientific system, but it seems that these duties are often not properly (if at all) recognized by universities, funding agencies or even the rest of the scientific community. This is the main message of the September Editorial in Nature Structural & Molecular Biology, ‘The Unsung Reviewer’ (16, 899; 2009) The Editorial notes:

“Scientists wear many hats these days. They prepare and teach classes and sit on various committees. Then there are the multiple activities directly related to their research, including grant writing, mentoring students and postdocs, attending conferences, writing papers and reviewing manuscripts. All these duties can take a considerable amount of time and effort and most are recognized as worthy contributions by funding agencies, universities and research institutions when evaluating a scientist’s performance. On the other hand, peer reviewing papers seems to be the Rodney Dangerfield (”I get no respect!“) of a scientist’s duties.”

A new report by the Science for Policy project of the US Bipartisan Policy Center notes that peer-review is essential for the scientific system. Journals such as NSMB (and the other Nature journals) would not agree from their own perspective with one of the report’s conclusions, that “peer-review is no longer assumed to be a professional obligation”. As the Editorial points out, our journals have very broad reviewer pools (running into many tens of thousands for the heavily oversubscribed weekly title Nature, for example), and the editors know and very much appreciate the work that these reviewers put into improving submitted manuscripts.

The SPP report also identifies a need to increase the number of scientists who participate in peer review for federal agencies, making suggestions such as listing such service on grant applications or even making it a requirement for funding. In addition, it urges journals to run a quality system, for example by providing the peer-reviewers with feedback (which the Nature journals do, and which is appreciated by our reviewers – for one reason, because a reviewer can see his or her report in the context of those of the other reviwers of the manuscript). The NSMB Editorial concludes:

“Peer review is often compared to jury duty, a chore that one has to do once in a while as a service to the community. But reviewing manuscripts can be enjoyable and useful at any career stage. True, it takes time, but as one reviewer aptly put it: “I still learn do’s and don’ts from reviewing papers, as well as just getting useful information, so it is time spent pretty well.” It is also true that one does not get much open recognition from it, and there are limitations on what the journals can do without compromising the reviewers’ anonymity. We have in the past publicly acknowledged our reviewers at the end of the year, and will do so again this year. The SPP report stresses that it is important to stress that “peer reviewing manuscripts should be an expected and appreciated aspect of a scientist’s career.” Of course, we do have a vested interest in this, but we firmly believe that it’s important to cultivate a vibrant ‘reviewer culture’."

Nature journals’ peer-review policies and guidelines.

More on peer review from the Nature journals.

Becoming a peer-reviewer for a journal

We received a question at the Nature Network ‘Ask the Editor’ forum which I thought readers of this blog might find useful.

Q: I once reviewed several manuscripts for a good journal, but it is on behalf of my Ph.D boss. Now, I am a postdoc and wonder how to become a reviewer? Thanks for any suggestions.

A. At Nature, we ask our peer-reviewers to identify anyone who helps them with their review. (The reviewers promise to keep the ms confidential, in advance of being sent it, and undertake to ensure that anyone they show it to also keeps it confidential). Therefore, we often discover (and regularly use) good new reviewers by this method. Many senior reviewers ask junior colleagues to review a ms as part of their mentoring, and are very good at assigning credit to these junior colleagues.

I suggest that if you help your boss or another colleague with his or her reviews in future, you ask that person to name you as a collaborator when he/she sends the journal the review, so your name gets known.

If you have been a co-author on your boss’s papers, you could contact the journal that published the work and offer to review (providing the information that you’ve published in that journal).

I think that as you publish in your own right, journal editors will get to know of you and start to ask you to review for them.

If you meet any journal editors at conferences you attend, you could let them know that you’d be interested in peer-reviewing for their journal.

Good luck!