Vaccination News Home Page

http://www.nytimes.com/2002/06/11/health/policy/11DOC.html

When Peer Review Yields Unsound Science

By LAWRENCE K. ALTMAN, M.D.

edical journals are the prime source of information about scientific advances that can change how doctors treat patients in offices and in hospitals. And to ensure the quality of what journals publish, their editors, beginning 200 years ago, have increasingly called on scientific peers to review new findings from research in test tubes and on animals and humans.

The system, known as peer review, is now considered a linchpin of science. Editors of the journals and many scientists consider the system's expense and time consumption worthwhile in the belief that it weeds out shoddy work and methodological errors and blunts possible biases by scientific investigators. Another main aim is to prevent authors from making claims that cannot be supported by the evidence they report.

Yet for all its acclaim, the system has long been controversial. Despite its system of checks and balances, a number of errors, plagiarism and even outright fraud have slipped through it. At the same time, the system has created a kind of Good Housekeeping Seal of Approval that gets stamped on research published in journals. Although most research is solid, and in some cases groundbreaking, problems have persisted.

A particular concern is that because editors and reviewers examine only what authors summarize, not raw data, the system can provide false reassurances that what is published is scientifically sound.

After a series of problems, about 20 years ago, journal editors came under pressure to better document claims for the system's merits. To do that, a number of editors began their own primary research into the way the peer review system worked, what was wrong and how it could be fixed.

A leader has been The Journal of the American Medical Association, which has held four meetings on research on peer review since 1989 under the direction of Dr. Drummond Rennie, a deputy editor. Last week, the journal published 34 articles from the latest meeting. And the news was grim.

Researchers reported considerable evidence that many statistical and methodological errors were common in published papers and that authors often failed to discuss the limitations of their findings. Even the press releases that journals issue to steer journalists to report peer reviewed papers often exaggerate the perceived importance of findings and fail to highlight important caveats and conflicts of interest.

"Once again," Dr. Rennie wrote in an editorial summarizing the findings, "we publish studies that fail to show any dramatic effect, let alone improvement, brought about by editorial peer review."

Under the system, authors submit manuscripts to journals whose editors send the most promising ones to other experts (peers) in academic medicine to solicit their unpaid advice. The peers check for obvious errors, internal inconsistencies, logic, statistical legitimacy, reasonableness of conclusions and many other factors, and make suggestions that editors use in deciding whether to ask for revisions, publish the paper or return it marked "rejected."

There is general agreement that an overwhelming majority of "weak" papers will survive initial rejections to find acceptance somewhere among the thousands of medical and scientific journals that each year publish an estimated two million new research articles, mostly paid for by the public through government grants. Despite improvements in peer review, "there still is a massive amount of rubbish" in the journals, Dr. Rennie said.

In recent years, editors have used the importance of peer review to justify imposing punitive restrictions on authors who disclose information to the press before the paper's publication in their journals. By linking peer review and publication date, critics say, editors have increased the news value of their journals, a step that has slowed the free flow of information and helped some journals raise subscriptions, advertisement rates and profits.

While many editors and others have defended the system, they acknowledge that it is unlikely to detect fraud and is prone to abuse.

One reason is that the secrecy involved in the system can be unfair to authors. While the names of authors are generally known to reviewers, the reviewers' names are not disclosed to the authors. Because the anonymous peers chosen to review manuscripts are often the authors' scientific competitors, jealousies and competitive advantage can become factors in the reviews.

Occasionally, reviewers have been caught publishing information they lifted from other researchers' manuscripts. Further, little is known about the quality of the reviewers or what training they need to do a good job.

"The available evidence," wrote Fiona Godlee of BioMed Central in London, "gives no indication that anonymous peer review achieves better scientific results than open review."

Apparently, few journals have adopted the open system. Ms. Godlee said she looked forward to the day when signed reviews were posted on the Internet along with published articles.

The peer review system also tends to set a very high barrier for authors to publish truly novel findings.

In 1796, a peer reviewed journal in England rejected Dr. Edward Jenner's report of his development of the world's first vaccine, against smallpox. The vaccine was used to eradicate the viral disease nearly two centuries later, and it may be needed again if bioterrorists release smallpox virus in an attack.

In recent decades, at least two Nobel Prizes were awarded to scientists who received rejection slips from one journal before another published their papers.

One paper concerned what turned out to be the hepatitis B virus. The other concerned a radio-immunoassay technique that can detect trace amounts of substances in the body and that is now used every day throughout the world.

Another recent problem, critics say, is that many editors have not moved quickly enough to use newer methods to judge the merits of manuscripts.

One example is the growing importance of statistics to measure the safety and effectiveness of new therapies and to compare them with older ones. Yet research on peer review has found that many studies are conducted without the benefit of adequate consultation with statisticians, sometimes because none were available.

Reasons for errors also include the practice of consulting statisticians after the research project has been completed, not at the most critical time, when the study was being designed.

"Expert analysis cannot salvage poorly designed research," wrote Dr. Douglas G. Altman of the Center for Statistics in Medicine in Oxford, England.

Once statistical errors are published, it is hard to stop them from spreading and being cited uncritically by others.

Dr. Richard Horton, editor of The Lancet, found in his own study that reviewers often did not detect important limitations to research findings that authors left out of their papers. The omission, Dr. Horton said, must be judged a failure of peer review.

Yet Dr. Rennie remained optimistic. Earlier research has led to some improvements. For example, in an interview Dr. Rennie said that findings from research in peer review had led some medical journals to adopt more standardized systems for reporting findings from the clinical trials that were used to determine the safety and effectiveness of new drugs and other therapies.

Also, because reviewers are selected for expertise the editors do not have, reviewers often rescue a paper from rejection.

For example, Dr. Rennie cited a scientist who found scientific merit in an author's manuscript that Dr. Rennie had sent him. The reviewer "completely rewrote" the manuscript that he "would have rejected otherwise," Dr. Rennie said.

No one knows how often such "rescues" occur, Dr. Rennie said, in part because the needed studies have not been done. Such studies would involve funds that have been hard to obtain as well as cooperation of reviewers, editors and authors of papers that were accepted and rejected, and many of them might not participate because of their set views.

In 1986, Dr. Stephen Lock, then editor of The British Medical Journal, wrote, "Editors, the arbiters of rigor, quality and innovativeness in publishing scientific work, do not apply to their own work the standards they apply to judging the work of others."

Defenders of the system still face Dr. Lock's challenge.

Vaccination News Home Page

ALL INFORMATION, DATA, AND MATERIAL CONTAINED, PRESENTED, OR PROVIDED HERE IS FOR GENERAL INFORMATION PURPOSES ONLY AND IS NOT TO BE CONSTRUED AS REFLECTING THE KNOWLEDGE OR OPINIONS OF THE PUBLISHER, AND IS NOT TO BE CONSTRUED OR INTENDED AS PROVIDING MEDICAL OR LEGAL ADVICE.  THE DECISION WHETHER OR NOT TO VACCINATE IS AN IMPORTANT AND COMPLEX ISSUE AND SHOULD BE MADE BY YOU, AND YOU ALONE, IN CONSULTATION WITH YOUR HEALTH CARE PROVIDER.