Collections under
which this article appears: Peer
Review
BMJ 2003;326:241 ( 1 February )
News
Little evidence for effectiveness of scientific peer review
Caroline White, London
Despite its widespread use and costs, little hard evidence exists that peer
review improves the quality of published biomedicalresearch,
concludes a systematic review from the internationalCochraneCollaboration.
Yet the system, which has been used for at least 200 years, has only recently
come under scrutiny, with its assumptions aboutfairness and
objectivity rarely tested, say the review authors.With few
exceptions, journal editorsand
cliniciansaround theworld continue to see it as the hallmark of serious scientific
endeavour.
Published last week, the review is the third in a series from the Cochrane
Collaboration Methods Group. The other reviewslook at the grant
application process and technicalediting.
Only the latter escapes a drubbing, with the reviewers concluding that
technical editing does improve the readability, accuracy,and overall
quality of publishedresearch.
The Cochrane reviewers based their findings on 21 studies of the peer review
process from an original trawl of only 135. Thesewere drawn from a
comprehensive search of biomedical print andonline databases, and
information received from bodies such asthe World Association of
MedicalEditors.
Almost half of the available research focused on the effects of concealing
the identity of reviewers and/or authors, which,the Cochrane authors
conclude, has little impact on quality. Fewstudies assessed the
impact of peer review on the importance,usefulness, relevance, or
quality of research. Only one smallstudy tested the validity of the
peer review procedureitself.
On the basis of the current evidence, "the practice of peer review is based
on faith in its effects, rather than on facts,"state the authors,
who call for large, government funded researchprogrammes to test the
effectiveness of the system and investigatepossible
alternatives.
"As the information revolution gathers pace, an empirically proven method of
quality assurance is of paramount importance,"they
contend.
Professor Tom Jefferson, who led the Cochrane review, suggested that further
research might prove that peer review, or anevolved form of it,
worked. At the very least, it needed to bemore open and
accountable.
But he said that there had never even been any consensus on its aims and that
it would be more appropriate to refer to itas "competitivereview."
Not only did peer review pander to egos and give researchers licence to knife
each other in the back with impunity, he said,but it was also
"completely useless at detecting research fraud"and let editors off
the hook for publishing poor qualitystudies.
In the latest report from the Committee on Publication Ethics, Professor
Peter Lachmann, until recently president of the UKAcademy of Medical
Sciences, comments: "Peer review is to sciencewhat democracy is to
politics. It's not the most efficient mechanism,but it's the leastcorruptible."
Footnotes
The report can be accessed from the National electronic Library for Health (www.nelh.nhs.uk)
ALL INFORMATION, DATA, AND
MATERIAL CONTAINED, PRESENTED, OR PROVIDED HERE IS FOR GENERAL INFORMATION
PURPOSES ONLY AND IS NOT TO BE CONSTRUED AS REFLECTING THE KNOWLEDGE OR OPINIONS
OF THE PUBLISHER, AND IS NOT TO BE CONSTRUED OR INTENDED AS PROVIDING MEDICAL OR
LEGAL ADVICE. THE DECISION WHETHER OR NOT TO VACCINATE IS AN IMPORTANT AND
COMPLEX ISSUE AND SHOULD BE MADE BY YOU, AND YOU ALONE, IN CONSULTATION WITH
YOUR HEALTH CARE PROVIDER.
"A foolish faith in authority is the worst enemy of truth."
-- Albert Einstein, letter to a friend, 1901
"I know of no safe depository of the ultimate powers of the society but the people themselves, and if we think them not enlightened enough to exercise control with a wholesome discretion, the remedy is not to take it from them, but to inform their discretion by education."
-- Thomas Jefferson, letter to William C. Jarvis, September 28, 1820
"What's the point of vaccination if it doesn't protect you from the unvaccinated?"
-- Sandy Gottstein
"Who gets to decide what the greater good is and how many will be sacrificed to it?"