http://www.economist.com/science/displayStory.cfm?Story_ID=779761

Publication ethics

Truth or consequences
Sep 13th 2001
From The Economist print edition

Biomedical journals are agitating about the risks of corporate money corrupting peer review. They should worry about a lot more than that

SOME of the most powerful words in print today are those in journals of biomedical research, for they can literally wield the power of life and death over future patients. If published clinical trials establish a drug's safety and efficacy, doctors will eventually prescribe it. If not, it will be relegated to the laboratory's hazardous-waste basket. Ensuring the accurate reporting of such trials is therefore a matter of some importance.

Unfortunately, maintaining the integrity of this sort of science seems to be getting ever more difficult. To select and revise promising papers, journal editors have long relied on the process of peer review, in which a manuscript is sent out for criticism to several experts in the field. If peer review is to work, each party must fulfil its side of the bargain. First, authors must genuinely have done the things they claim to have done, and written the documents they claim to have written. Second, reviewers must show that they are able to assess the research as objectively as possible.

It appears that, in too many cases, neither side of the bargain is being kept. This weekend, several hundred researchers and medical journalists are supposed to be gathering in Barcelona for a conference organised by the World Association of Medical Editors on the effects of peer review on biomedical research. Some of the findings to be presented are damning of researchers, of reviewers and of journals themselves. Although, to their credit, editors of medical journals are seizing every opportunity to tighten the reins of control over biomedical research, many of the problems defy any quick fix.

Quis custodiet ipsos custodes?

The issue topping the editorial agenda is the honesty of research that is sponsored by drug companies to test the safety of their wares. These companies have found that scientists at private contract-research firms can perform drug trials far more cheaply than the academic researchers who have traditionally been drafted in for this job. Such firms drew 60% of the research funds spent by drug companies last year; academic scientists were left scrabbling for the rest.

Unfortunately, the companies are saving more than money by using such hired guns; they are also saving face. Through the use of restrictive research agreements, drug companies often ensure that only favourable results are published. In some cases, they not only retain the right to suppress publication of results, they also deny the authors of papers about such studies access to full sets of study data.

In the face of this, such authors cannot give honest accounts of the research; nor can editors and reviewers make honest appraisals of the results. This week, the editors of 13 top medical journals issued a statement condemning such abuses of corporate sponsorship. The members of the International Committee of Medical Journal Editors (ICMJE) will now revise their publication guidelines to require that researchers disclose their sponsors' involvement to editors when they submit articles. Some editors will require researchers to certify that they take responsibility for how the trial they are reporting was conducted. And authors may be asked to declare that they had access to all trial data, and had the power to decide whether or not to publish the trials' results.

In a sense, these rules are formal demands for things that ought to be taken for granted. But, as a paper due to be presented at Barcelona demonstrates, it is not even clear that authors pay enough attention to the existing ICMJE guidelines, let alone that they will be willing to abide by extra ones.

The existing guidelines state that people on a paper's author list should have contributed, at the very least, to the conception of a study, the analysis of the data and the writing of the manuscript. Yet, when Susan van Rooyen and Sandra Goldbeck-Wood of the British Medical Journal and Fiona Godlee of BioMed Central, a database of scientific abstracts, analysed 129 research articles submitted to the BMJ by 588 authors, they found that this was not always the case. Depending on how strictly they interpreted the ICMJE guidelines, between 24% and 71% of authors qualified—hardly a stellar performance.

Richard Horton, the editor of the Lancet, took an even looser definition of authorship. He studied a small sample of research papers to find out if the views expressed in those articles were accurate representations of contributors' opinions—a basic enough requirement, one might think. He examined in detail five papers published in the Lancet last year, and found that contributors disagreed about almost everything, including a paper's key findings, weaknesses, implications and future prospects.

That may not surprise anybody who has ever worked on a team project. Dr Horton, however, also sniffed out a more disturbing trend. The contributor who stood to gain the most credit from a publication would often erase from the final report the weaknesses acknowledged by co-contributors. A research paper rarely represents the opinions of all the scientists whose work it reports, Dr Horton concludes.

Ideally, the “suppressed opinion, censored criticism and serious bias” among contributors found by Dr Horton would be corrected by the unfettered opinions and criticism offered by unbiased reviewers. But authors may obstruct reviewers' judgments by leaving out important information about their results, such as details of other similar trials. According to research conducted by the Cochrane Collaboration, an international group that studies clinical trials, reviewers do not seem to be up to catching these omissions. Indeed they may be getting worse at their task.

In May 1997, a group of Cochrane researchers, led by Iain Chalmers, investigated whether reports of trials published in the five top medical journals adequately outlined research on other relevant trials, by studying one month's worth of reports. Almost none of the 26 reports passed the test. This year, the team repeated the procedure, using the 33 trials published by these five journals in May. None of the trials put its findings in the context of earlier trials. Four reports falsely stated that they were the first published trial on a particular topic. Six more kept mum about related trials. Dr Chalmers and his colleagues find that the situation has deteriorated in the past four years.

This is strong stuff. But it may get worse. Phil Alderson and his colleagues performed a “meta-analysis” of editorial peer review—that is, they used published studies as their data, and applied statistical methods to the outcomes of those studies, as though each study were a single experimental result. They looked at the effects of peer review on various criteria, including methodological soundness, completeness and accuracy.

Although journal editors go to great pains to ensure that authors do not know the identity of their reviewers, or vice versa, the researchers found that this laborious and expensive process had little impact on the reviewers' appraisals of quality of research. When they surveyed the sum of research on peer review, they found only scattered empirical evidence supporting the use of editorial peer review as a mechanism to ensure quality of biomedical research. Until better studies are conducted, peer review may need to undergo more, well, peer review.

ALL INFORMATION, DATA, AND MATERIAL CONTAINED, PRESENTED, OR PROVIDED HERE IS FOR GENERAL INFORMATION PURPOSES ONLY AND IS NOT TO BE CONSTRUED AS REFLECTING THE KNOWLEDGE OR OPINIONS OF THE PUBLISHER, AND IS NOT TO BE CONSTRUED OR INTENDED AS PROVIDING MEDICAL OR LEGAL ADVICE.  THE DECISION WHETHER OR NOT TO VACCINATE IS AN IMPORTANT AND COMPLEX ISSUE AND SHOULD BE MADE BY YOU, AND YOU ALONE, IN CONSULTATION WITH YOUR HEALTH CARE PROVIDER.