The Emperor's New Drugs: An Analysis of Antidepressant Medication Data Submitted to the U.S. Food and Drug Administration

Finally, in many of the trials reported to the FDA, various fixed doses of the active medication were evaluated in separately randomized arms. Finding a dose-response relationship is one method of establishing the presence of true drug effects. Also, a dose-response relationship suggests that the drug effect may be underestimated in trials involving low dosages. Therefore, our analyses include a comparison of treatment effects at the lowest doses employed in fixed-dose trials with those at the highest doses.

Method

Using the Freedom of Information Act, we obtained the medical and statistical reviews of every placebo controlled clinical trial for depression reported to the FDA for initial approval of the six most widely used antidepressant drugs approved within the study period. We received information about 47 randomized placebo controlled short-term efficacy trials conducted for the six drugs in support of an approved indication of treatment of depression. The breakdown by efficacy trial was as follows: fluoxetine (5), paroxetine (16), sertraline (7), venlafaxine (6), nefadozone (8), and citalopram (5). Data on relapse prevention trials were not analyzed.

In order to generalize the findings of the clinical trial to a larger patient population, FDA reviewers sought a completion rate of 70% or better for these typically 6-week trials. Only 4 of 45 trials, however, reached this objective. Completion rates were not reported for two trials. Attrition rates were comparable between drug and placebo conditions. Of those trials for which these rates were reported, 60% of the placebo patients and 63% of the study drug patients completed a 4-, 5-, 6-, or 8-week trial. Thirty-three of 42 trials lasted 6 weeks, 6 trials lasted 4 weeks, 2 lasted 5 weeks, and 6 lasted 8 weeks. Patients were evaluated on a weekly basis. For the present meta-analysis, the data were taken from the last visit prior to trial termination.

Although the FDA approved the drugs for "the treatment of depression" not otherwise specified, all but one of the clinical trials were conducted on patients described as moderately to severely depressed (their mean baseline Hamilton Depression Scale [HAM-D] scores ranged from 21.0 to 29.7). One of the trials was conducted on patients with mild depression (mean baseline HAM-D score = 17.21). Thirty-nine of the 47 clinical trials focused on outpatients, 3 included both inpatients and outpatients, 3 were conducted with elderly patients (including one of the trials with both inpatients and outpatients), and 2 were conducted among patients hospitalized for severe depression. No trial was reported for the treatment of children or adolescents.

After 2 weeks, replacement of patients was allowed for those who investigators determined were not improving in three fluoxetine trials and in the three sertraline trials for which data were reported. The trials also included a 1- to 2-week placebo washout period, during which patients were given placebo. Those whose scores improved 20 percent or more were excluded from the study. The use of other psychoactive medication was reported in 25 trials. In most trials, a chloral hydrate sedative was permitted in doses ranging from 500 mg to 2000 mg per day. Other psychoactive medication was usually prohibited but still was reported as having been taken in several trials.

For each clinical trial, we recorded the mean improvement in HAM-D scores in the drug and placebo groups. Next, improvement in the placebo group was divided by improvement in the drug group to provide an estimate of the degree of improvement in the drug-treated patients that was duplicated in the placebo group. Then, the mean of each of these trials, weighted for sample size, was calculated within each drug.

Results

The 17-item version of the HAM-D was used in all trials of paroxetine, sertraline, nefazodone, and citalopram. The 21-item version was used in trials of fluoxetine and venlafaxine. One citalopram trial reported scores on both the 17-item scale and the 21-item scale, and another reported scores on the 17-item scale and a 24-item version of the scale. We used the 17-item scores for citalopram studies because this version of the scale was used in all of the clinical trials of that medication. Calculation of response to drug and placebo for the two studies using different forms of the scale reveals that the drug/placebo comparison is comparable, regardless of which scale is used.

Mean improvement scores were not reported in 9 of the 47 trials. Specifically, four paroxetine trials involving 165 participants, four sertraline trials involving 486 participants, and one citalopram trial involving 274 participants were reported as having failed to achieve a statistically significant drug effect, but the mean HAM-D scores were not reported. This represents 11% of the patients in paroxetine trials, 38% of the patients in sertraline trials, and 23% of the patients in citalopram trials. In each case, the statistical or medical reviewers stated that no drug effect was found.

Including data from paroxetine and sertraline trials in summary statistics would produce an inflated estimate of drug effects. Therefore, to obtain an unbiased estimate of drug and placebo effects across medications, we calculated weighted means of all medications for which data on all clinical trials were reported. This included the data for fluoxetine, venlafaxine, and nefadozone. The weighted mean difference between the drug and placebo groups across these three medications was 1.80 points on the HAM-D, and 82% of the drug response was duplicated by the placebo response. A t-test, weighted for sample size, indicated that the drug/placebo difference was statistically significant, t(18) = 5.01, p < .001.

On most of the clinical trials, medication dose was titrated individually for each patient within a specified range. However, in 12 trials involving 1,942 patients, various fixed doses of a medication were evaluated in separately randomized arms. It is possible that some of the doses used in these trials were subclinical. If this is the case, inclusion of these data could result in an underestimate of the drug effect. To test this possibility, we compared LOCF data at the lowest and highest doses reported in each study. Across these 12 trials, mean improvement (weighted for sample size) was 9.57 points on the HAM-D at the lowest dose evaluated and 9.97 at the highest dose. This difference between high and low doses of antidepressant medication was not statistically significant.

Finally, we tested the hypothesis that LOCF analyses provide more conservative tests of drug effects than do OC analyses. LOCF means were reported for all 38 of the 46 trials in which means of any kind were reported. OC means were reported for 27 of these 38 trials. In 22 trials, the difference between drug and placebo group was not statistically significant with either LOCF or OC measures. In 12 trials, the difference was statistically significant with both measures. In 8 trials, the difference was significant with LOCF but not with OC, and 4 trials were reported to have shown no difference between drug and placebo without specifying an attrition rule. For the 27 trials for which both sets of means were reported, correlated t-tests indicated that mean improvement scores were significantly greater with OC data than with LOCF data for both drug, t(26) = 12.46, p < .001, and placebo, t(26) = 10.56, p < .001, as was the proportion of the drug response duplicated by placebo, t(26) = 3.36, p < .01. In the LOCF data, 79% of the drug response was duplicated in the placebo groups; in the OC data, 85% of the drug response was duplicated by placebo. Thus, LOCF analyses indicate a greater drug/placebo difference than do OC analyses.

Discussion

Of the two widely used methods of coping with attrition in clinical trials, LOCF analyses are considered the more stringent. The FDA data set calls this assumption into question. The proportion of the drug effect duplicated by placebo was significantly larger in the OC data set than in the corresponding LOCF data set. In addition, the degrees of freedom are necessarily larger in LOCF analyses, thereby making it more likely that a mean difference will be statistically significant. In the 47 clinical trials obtained from the FDA, there were no reported instances in which OC data yielded significant differences that were not detected in LOCF analyses. However, in 8 trials, LOCF data yielded significant differences that were not detected when OC data were analyzed. These data indicate that, compared with LOCF analyses, OC analyses provide more conservative tests of drug/placebo differences.

If antidepressant drug effects and antidepressant placebo effects are not additive, the ameliorating effects of antidepressants might be obtained even if patients did not know the drug was being administered. If that is the case, then antidepressant drugs have substantial pharmacologic effects that are duplicated or masked by placebo. In this case, conventional clinical trials are inappropriate for testing the effects of these drugs, as they may result in the rejection of effective medications. Conversely, if drug and placebo effects of antidepressant medication are additive, then the data clearly show that those effects are small, at best, and of questionable clinical efficacy. Finally, it is conceivable that the effects are partially additive, with the true drug effect being somewhere in between these extremes. The problem is that we do not know which of these models is most accurate because the assumption of additivity has never been tested with antidepressant mediation.

Depending on assignment, participants are (a) told they are getting the drug and do in fact receive it, (b) told they are getting drug but in fact receive placebo, (c) told they are getting placebo but in fact receive drug, and (d) told they are getting placebo and in fact receive placebo. This permits independent and combined assessment of drug and placebo effects.

Without the assumption of additivity, the FDA data do not allow one to determine the effectiveness of antidepressant medication. That is, it is not possible to determine the degree to which the antidepressant response is a drug effect and the degree to which it is a placebo effect. If one does make the assumption that the drug effect is the difference between the drug response and the placebo response, then it is very small and of questionable clinical value. By far, the greatest part of the change is also observed among patients treated with inert placebo. The active agent enhances this effect, but to a degree, that may be clinically meaningless.

These data raise questions about the criteria used by the FDA in approving antidepressant medications. The FDA required positive findings from at least two controlled clinical trials, but the total number of trials can vary. Positive findings consist of statistically significant drug/placebo differences. The clinical significance of these differences is not considered.

To summarize, the data submitted to the FDA reveal a small but significant difference between antidepressant drug and inert placebo. This difference may be a true pharmacological effect, or it may be an artifact associated with the breaking of blind by clinical trial patients and the psychiatrists who are rating the severity of their conditions. Further research is needed to determine which of these is the case.

In any case, the difference is relatively small (about 2 points on the HAM-D), and its clinical significance is dubious. Research is therefore needed to assess the additivity of antidepressant drug and placebo effects. If there is a powerful antidepressant effect, then it is being masked by a nonadditive placebo effect, in which case current clinical trial methodology may be inappropriate for evaluating these medications, and alternate methodology need to be developed. Conversely, if the drug effect is as small as it appears when drug/placebo differences are estimated, then there may be little justification for the clinical use of these medications. The problem, then, would be to find an alternative, as the clinical response to both drug and placebo is substantial. Placebo treatment has the advantage of eliciting fewer side effects. However, the deception that is inherent in clinical administration of placebos inhibits their use. Thus, the development of nondeceptive methods of eliciting the placebo effect would be of great importance.

References

Fisher, S., & Greenberg, R. P. (1993). How sound is the double-blind design for evaluating psychotropic drugs. Journal of Nervous and Mental Disease, 181, 345-350.

Hamilton, M. A. (1960). A rating scale for depression. Journal of Neurology, Neurosurgery, and Psychiatry, 23, 56-61.

Hull, J. G., & Bond, C. F. (1986). Social and behavioral consequences of alcohol consumption and expectancy: A meta-analysis. Psychological Bulletin, 99, 347 360.

Hunter, J. E., & Schmidt, F. L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.

Khan, A., Warner, H. A., & Brown, W. A. (2000). Symptom reduction and suicide risk in patients treated with placebo in antidepressant clinical trials: An analysis of the Food and Drug Administration database. Archives of General Psychiatry 57, 311-317.

Kirsch, I. (2000). Are drug and placebo effects in depression additive? Biological Psychiatry 47, 733-73.

Kirsch, I., & Rosadino, M. J. (1993). Do double-blind studies with informed consent yield externally valid results? An empirical test. Psychopharmacology, 110, 437-442.

Laughren, T. P. (1998, March 26). Recommendation for approvable action for Celexa (citalopram) for the treatment of depression. Memoradum: Department of Health and Human Services, Public Health Service, Food and Drug Administration, Center for Drug Evaluation and Research, Washington, DC.

Leber, P. (1998, May 4). Approvable action on Forrest Laboratories, Inc. NDA 20-822 Celexa (citalopram HBr) for the management of depression. Memoradum: Department of Health and Human Services, Public Health Service, Food and Drug Administration, Center for Drug Evaluation and Research, Washington, DC.

Lyerly, S. B., Ross, S., Krugman, A. D., & Clyde, D. J. (1964). Drugs and placebos: The effects of instructions upon performance and mood under amphetamine sulphate and chloral hydrate. Journal of Abnormal and Social Psychology, 68, 321 327.

Marlatt, G. A., & Rohsenow, D. J. (1980). Cognitive processes in alcohol use: Expectancy and the balanced placebo design. In N. K. Mello (Ed.), Advances in substance abuse: Behavioral and Biological Research, (pp. 159 199). Greenwich, CT: JAI Press.

Murray, E. J. (1989). Measurement issues in the evaluation of pharmacological therapy. In S. Fisher & R. P.Greenberg (Eds), The limits of biological treatments for psychological distress: Comparisons with psychotherapy and placebo (pp. 39-67). Hillsdale, NJ: Erlbaum.

Rabkin, J.G., Markowitz, J. S., Stewart, J. W., McGrath, P. J., Harrison, W., Quitkin, F. J., & Klein, D. F. (1986) How blind is blind? Assessment of patient and doctor medication guesses in a placebo-controlled trial of imipramine and phenelzine. Psychiatry Research, 19, 75-86.

Ross, S., Krugman, A. D., Lyerly, S. B., & Clyde, D. J. (1962). Drugs and placebos: A model design. Psychological Reports, 10, 383 392.