How do we know that any treatment for mental illness works?
The same as in the rest of Medicine – from the results of randomised controlled trials.
The simplest way to see if a treatment works is just to give it to some patients. An uncontrolled trial can estimate whether a treatment works at all and give a rough idea of adverse effects. Most new treatments go through this stage, but it is an unreliable guide. Basically, any treatment given or taken with enough enthusiasm can have benefits (and vice-versa). And, any benefit or harm from the treatment could be due to spontaneous improvement or worsening in the condition being treated. A group of controls is needed to allow for such possible placebo and non-specific effects. Hormone injections, tooth removal and other gruesome historical aspects of psychiatry (and the rest of medicine) might have been avoided if the need for controlled trials had been appreciated earlier. 
A ‘control group’ in a clinical trial is a similar group of patients who are ideally treated identically other than with the treatment under investigation – to see if the new treatment works better than placebo or a pre-existing treatment. The problem is that new treatments tend to be developed and studied by enthusiasts and so patients who get a new therapy tend to be selected – consciously or unconsciously – to be less ill than those who get the standard old treatment. Such ‘selection bias’ of good prognosis patients can seriously overestimate the benefits of new therapeutic approaches. This is why we need randomisation.
For example, it was only when a randomised, controlled trial of insulin coma therapy (ICT) was done that psychiatrists fully realised it didn’t work. Patients were either given insulin to induce coma or with unconsciousness produced by barbiturates. There was no difference in outcome between the groups and ICT fell out of use.
A randomised controlled trial is one in which all patients have an equal chance of getting whatever treatments are being tested. Randomisation, if carried out properly, rules out the possibility that selection bias (conscious or unconscious) could account for the results of the study. It also accounts for other patient factors that could confound the results. This is why randomised controlled trials are held in such high regard.
It is however a critical part of randomisation that people do not know what treatment they will get at the point that they consent. This is called ‘allocation concealment’. If anyone involved does know or can work out what treatment the next person in the trial is getting, selection bias tends to creep in again and the trial will typically overestimate the benefits of treatment by about 30%, which is more than the benefit of many treatments in medicine.
The other big potential problem in clinical trials is that people giving the treatment, or getting it, or those rating how well the patients did, could be influenced by knowing which treatment they got. This could mean that the patient gets the treatment more enthusiastically or with other benefits, or expect to get better more, and are rated as such. On average, an adequately blinded trial will find a treatment is 17% less effective than a non-blinded trial, although it varies depending on the outcome measure. A ‘hard’ outcome like death is less prone to bias than ‘soft’ outcomes like symptom severity and quality of life. This is therefore a particular problem in mental illness trials and especially for procedures like psychotherapy which cannot be given without the therapist knowing they are giving it and the patient knowing they are getting it.
This situation demands rigorously blinded outcome raters -and even they could work out what patients were getting from side effects of drugs or the patient simply saying they got psychotherapy of one sort or another. One rather time-consuming and expensive solution to this problem is to record interviews about outcome and edit out any such clues before the final outcome rating.
One trial is (almost) never enough
Just as one swallow does not make a summer, one study can hardly ever be so well conducted and get such clear results as to prove anything. This is why researchers place such an emphasis on replication i.e. two or more studies finding the same thing. And once there are more than a couple of studies they can disagree. This is where systematic reviews and meta-analyses come in…(see a later blog).
Fortunately, but mainly because the treatments work, landmark randomised controlled trials (RCTs) of antipsychotic drugs for schizophrenia, lithium for bipolar disorder, and antidepressants or CBT for depression, have been widely replicated and have convincing benefits. They both get people better and – if maintained – reduce relapse rates for two years or so.[4,5] These relatively specific effects of particular treatments for certain conditions justify our major diagnostic categories.
It is through the careful application of RCTs and earlier diagnosis that the outcome of cancer has improved so much in recent years. The same could happen for mental illness if we could do more clinical trials….
 A. Scull, Madhouse: A Tragic Tale of Megalomania and Modern Medicine. New Haven and London: Yale University Press, 2005.
 Ackner B, Harris A, Oldham AJ. Insulin treatment of schizophrenia; a controlled study. Lancet. 1957 Mar 23;272(6969):607-11.
 Jüni P, Altman DG, Egger M. Systematic reviews in health care: Assessing the quality of controlled clinical trials. BMJ. 2001 Jul 7;323(7303):42-6. http://www.ncbi.nlm.nih.gov/pubmed/11440947
 Huhn M, Tardy M, Spineli LM, Kissling W, Förstl H, Pitschel-Walz G, Leucht C, Samara M, Dold M, Davis JM, Leucht S. Efficacy of pharmacotherapy and psychotherapy for adult psychiatric disorders: a systematic overview of meta-analyses. JAMA Psychiatry. 2014 Jun;71(6):706-15. http://www.ncbi.nlm.nih.gov/pubmed/24789675
Prepared for International Clinical Trials Day (#ClinicalTrialsDay) day on Wed 20th May 2015.