Review queries usefulness of peer review

Cochrane Review finds little evidence that peer-review ensures quality of biomedical research. | By Pat Hagan
 

The future of the current system of peer review has again come under question, after a report concluded there is little evidence to show it upholds good science.

The review, "Editorial Peer Review for Improving the Quality of Reports Of Biomedical Studies," was published last week by The Cochrane Collaboration, a UK body that releases regular systematic reviews of medical and scientific evidence.

After analysing 135 studies to measure the effects of peer review, the reviewers concluded: "At present, there is little empirical evidence to support the use of editorial peer review as a mechanism to ensure quality of biomedical research, despite its widespread use and costs. A large, well-funded programme of research on the effects of editorial peer review is needed."

The reviewers are keen to stress that lack of evidence that peer-review works is not the same as evidence that it doesn't work. They emphasize, for example, that the system does make papers more readable and may improve the general quality of reporting.

But the conclusions add to the growing debate among editors of leading medical and scientific journals over whether existing peer review is in fact damaging, rather than enhancing, the credibility of research.

"If peer review was a new drug it would not be approved," said one of the Cochrane reviewers, Thomas Jefferson of Rome-based firm Health Reviews Ltd. "But I don't think it should be shot at dawn. What we should be doing is more research into it.

"We know that technical editing does make a difference to the readability and we know that copy editing is important to pick up on contrasting statements.

"But we may want to fund a review on the general application of peer review."

Most of the criticism surrounding peer review has centred on its inability to detect flawed research, the secrecy involved in reviewing submitted papers, and the lack of transparency in the whole process.

Anonymous peers can reject or unfairly criticize papers from direct competitors. In the very worst cases, it's even been known for them to publish data "lifted" from a rival's work.

But the criticism also extends to the ability of peers to judge the quality of new work. If they get it wrong and shoddy or fraudulent work gets published, the potential for that misleading information to be used over and over again is disturbing.

British Medical Journal editor Richard Smith agrees with the Cochrane conclusion that peer review needs changing.

"The more we study peer review the more we find evidence of its deficiencies but we have no good evidence of its benefits. It slows things down and is very expensive. And if you ask two people whether something should be published, you'll get two different answers."

Smith and his BMJ team now run a system where reviewers must make their identity known to authors. They are also running a trial where reviewers' comments — and authors' responses — are published on the web, the results of which are likely to be published later this year or early in 2004.

"It will turn peer review from being a closed box, where you put something in at one end and wait to see what comes out at the other, to a scientific discourse that goes on in front of people."

Smith says there has already been a culture change among reviewers. Five yeas ago, a BMJ straw poll suggested half would decline to assess papers if forced to declare their identity. These days it's 5% or less.

But although there seems to be support for changes among those in medical research, in other scientific disciplines resistance is strong.

Donald Kennedy, editor of Science magazine, argued in an editorial last July that peer review has never provided "immunity against clever fraud." Prevention of misconduct, he believes, can only be achieved through teaching young scientists about the importance of honesty and trust.

Peter Newmark, editorial director at the open-access journal publisher BioMed Central (The Scientist's partner), says feedback from those in biology suggests they are not ready for such a big change. The BioMed Central (BMC) medical journals currently use a system of open peer review, where two named reviewers vet each paper and their reviews are posted along with the accepted paper. But among the 18 journals in the BMC biology series, reviewers remain anonymous.

A BMC survey of past biology authors and reviewers revealed 63% of reviewers were in favour of remaining anonymous. Perhaps more surprisingly, a similar number of authors felt the same. "I can only assume they were thinking they were going to be a reviewer at some point as well," said Newmark.

He believes a potential failing of open peer review is that reviewers will be more guarded in their criticisms. This may be because the authors are friends or colleagues, or that one day they could be pulling the purse strings on research grants.

"It's very hard to know if open peer review will have a positive effect," says Newmark. "It could lead to poor studies being published more easily because people will not be prepared to criticize as much as they would otherwise."