Observations on the report Evidence Check 2: Homeopathy by the House of Commons Science and Technology Committee, February 2010
1. Background
1.1. The report Evidence Check 2: Homeopathy was the second to be produced with the purpose of examining how the UK Government uses evidence to formulate and review its policies. It was not an inquiry into homeopathy as such. The House of Commons Committee asked two principal questions: What is the Government’s policy? And on what evidence is that policy based? The point was whether the scientific evidence supported the provision of homeopathy by the NHS and the licensing of homeopathic products by the MHRA.
1.2. The report received much publicity because of its firm rejection of evidence for homeopathy’s efficacy on its way to answering these questions. The aim of this paper is to focus on this one aspect of the Committee’s work, in view of doubts voiced about the validity of its findings. Sections 2 – 5 below address this question.
1.3. The author served on the House of Lords Science and Technology Sub-Committee which in 1999-2000 inquired into complementary and alternative medicine (CAM). He was Co-Chairman of what used to be called the Parliamentary Group for Alternative and Complementary Medicine during the 1990s, and also served on the advisory board to the systematic review of water fluoridation which was conducted in 1999-2000 by the NHS Centre for Reviews and Dissemination (CRD) at the University of York. As a user of homeopathy he has failed to derive much benefit from it, but has supported its use and development in the UK.
2. The scientific evidence for efficacy
2.1. There have been a number of systematic reviews and meta-analyses in this field, which as the Committee states are the best sources of evidence. The most recent review of substance is that by Shang et al in 2005, which it considered “the most comprehensive to date” and which compared 110 placebo-controlled trials of homoeopathy [authors’ spelling] with 110 trials of conventional medicine matched for disorder and type of outcome. The Committee cited a conclusion by the authors [paragraph 69] that “when analyses were restricted to large trials of higher quality there was no convincing evidence that homeopathy [sic] was superior to placebo”. They did not also cite the authors’ interpretation which followed these findings in the Lancet summary, which stated: “When account was taken for these biases [common to trials of both homoeopathy and conventional medicine], there was weak evidence for a specific effect of homoeopathic remedies, but strong evidence for specific effects of conventional interventions. This finding is compatible with the notion that the clinical effects of homoeopathy are placebo effects.”
2.2. This was no endorsement of homeopathy. But it was some way removed from the Committee’s conclusion in paragraph 70 of their report, “In our view, the systematic reviews and meta-analyses conclusively demonstrate that homeopathic products perform no better than placebos.” It also provides little support for that part of Professor Ernst’s evidence to the Committee where he “pointed out that: . . . Shang et al very clearly arrived at a devastatingly negative overall conclusion” [67].
2.3. The exaggeration by the Committee of Shang’s conclusions is worrying. It is difficult to see how a weakly supported positive effect, for which one explanation (possibly well-founded) is a placebo effect, can be translated into a conclusive demonstration of this effect, with a “devastatingly” negative finding. No such firm claims can be found in Shang, who writes of finding “no strong” evidence, or “little” evidence, and who ends his paper with cautions about methodology and about the difficulty of detecting bias in studies, as well as the role of possible “context effects” in homeopathy.
2.4. The Committee’s overstatement is not helped by claiming Government support for its interpretation in paragraph 70, based on the Minister’s concession of no “credible” evidence that homeopathy works beyond placebo. If he meant persuasive evidence - and his guarded support for further research [75] supports this - that shows a confusion by the Committee between absence of evidence and evidence of absence. If however he was saying that all evidence was negative, this as Prof. Harper correctly stated [71] runs counter to the message from most reviews up to and including Shang, which is one of primary studies of insufficient quantity, rigour, size, homogeneity and power to give clear-cut answers.
2.5. It is the absence of reliable evidence that remains the problem, and this includes evidence of an absence of specific effects (while acknowledging the problem in proving a negative, an obstacle which did not deflect the Committee from its conclusive verdict in 70). The Committee itself writes in 69 of no “convincing” evidence from Shang, from higher-quality trials, which is not consistent with a claim of conclusive (dis)proof. Care with words can be as important as with figures, and can just as easily mislead.
2.6. In a search for best evidence in the early 2000s this author relied on the bulletin on homeopathy produced by the NHS CRD at York in 2002, one of an Effective Health Care series on “the effectiveness of health service interventions for decision makers”. This bulletin made a systematic assessment of the evidence to date. It advised “caution” in interpreting this evidence, and warned that many of the areas researched were “not representative of the conditions that homeopathic practitioners usually treat”, and that “the methodological problems of the research” should be considered. It found “insufficient evidence of effectiveness . . to recommend homeopathy for any specific condition”. At the same time it could not conclude that homeopathy performed no better than placebo.
2.7. That was eight years ago. But it is notable that the more recent review by Shang, on which the Committee relied quite heavily, cited no reference to any placebo-controlled trial (i.e. of reasonable quality) subsequent to the CRD’s bulletin, in arriving at a suggestion, but not a conclusion, of a placebo effect. The House of Commons Committee’s verdict in 70 stands on its own in going beyond what either review found from the evidence before it.
2.8. In seeking an up-to-date assessment from the NHS CRD, this author was referred to the German researcher Klaus Linde as among the best of the objective sources of current evidence on homeopathy. Linde, who was the lead author of a major review in 1997 cited by the Committee, in turn recommended the statistician Rainer Lüdtke as an expert with a good overview of the current literature. Correspondence ensued with both researchers, who were aware of the Committee’s recent report.
2.9. Both Linde and Lüdtke hold that the Committee’s conclusion in 70 that reviews “conclusively demonstrate” a placebo effect is overstated and unsustainable on present evidence. They have further criticisms of the way in which evidence has been addressed.
2.10. Both are critical of Prof. Ernst’s evidence to the Committee as highlighted in 67. Prof. Linde confirms that his own 1999 re-analysis weakened the findings of his 1997 review and probably “at least overestimated the effects of homeopathic treatments”, but that his paper was “not ‘negative’” as stated by Ernst. He writes that “A more accurate interpretation is that the ‘re-analyses’ [by himself and 5 others, referred to by Ernst] show that the (positive) evidence is not fool-proof. This applies still today (for example, to the Shang analysis)”. Lüdtke draws attention to his own paper in 2002 which criticised many statistical errors in Ernst’s 2000 re-analysis in the same journal, vitiating Ernst’s negative conclusion, a published criticism which received no mention in Ernst’s own evidence to the Committee. Ernst was correct to state in evidence elsewhere that the re-analyses of Linde came to a “less than positive” conclusion, and that further reviews “failed to conclude that homeopathy is effective”. The Committee, while adopting Ernst’s more absolute conclusions, has not resolved the contradiction between his statements.
2.11. Lüdtke, like Shang, has also drawn attention to the pitfalls in research into homeopathy, in a chapter in ‘New directions in homeopathy research’ (Witt C, Albrecht H, eds.) published in 2009. He counsels against including all types of homeopathy trials of reasonable quality in one review (such reviews tend to suggest that homeopathic medicines are not efficacious), since the pooling of so many different kinds of trial and type of homeopathy makes findings unreliable. He advocates restricting systematic reviews to clearly defined health conditions or to single homeopathic medicines, concluding that “the heterogeneity of trials is high and the meta-analysis results are not robust against small changes in study design or statistical analysis”. In a paper published in 2008 he has argued that Shang’s conclusions do not hold when slightly different selection criteria are applied, e.g. by redefining how large is a “large” study, or by including treatment trials but excluding prevention trials. Size is not the only factor in arriving at robust conclusions.
2.12. Context effects may play a part, according to both Shang and Lüdtke. Shang’s “powerful alliances” between patient and carer, based on “shared strong beliefs”, may not be as distinctive or as peculiar to homeopathy as the nature of the homeopathic consultation, with its wider range of questions than are addressed in a conventional context, and the lifestyle recommendations referred to by Lüdtke that often flow from it. There is overlap here with the placebo effect (see 4 below); but homeopathy as “a complex medical system of its own” may be responsible for some broader effects.
2.13. Linde writes that the “undecided fraction” to which he belongs is confused by “the notorious lack of predictable reproducibility” on the one side, and by “too many anomalous results in high quality studies to rule out a relevant phenomenon” on the other.
more…
