Observations on the report Evidence Check 2: Homeopathy by the House of Commons Science and Technology Committee, February 2010

 1.  Background

 1.1.  The report Evidence Check 2: Homeopathy was the second to be produced with the purpose of examining how the UK Government uses evidence to formulate and re­view its policies.   It was not an inquiry into homeopathy as such.   The House of Commons Committee asked two principal questions:  What is the Government’s policy?   And on what evidence is that pol­icy based?   The point was whether the scientific evid­ence supported the pro­vision of homeopathy by the NHS and the licensing of homeo­pathic products by the MHRA.

 1.2.  The report received much publicity because of its firm rejection of evidence for homeopathy’s efficacy on its way to answering these questions.   The aim of this paper is to focus on this one aspect of the Committee’s work, in view of doubts voic­ed about the validity of its findings.   Sections 2 – 5 below address this question.

 1.3.  The author served on the House of Lords Science and Technology Sub-Commit­tee which in 1999-2000 inquired into complementary and alternative medicine (CAM).   He was Co-Chairman of what used to be called the Parlia­mentary Group for Alternative and Complementary Medicine during the 1990s, and also served on the advisory board to the systematic review of water fluoridation which was con­ducted in 1999-2000 by the NHS Centre for Reviews and Dissemin­ation (CRD) at the Uni­vers­ity of York.   As a user of homeopathy he has failed to derive much benefit from it, but has supported its use and development in the UK.

 2.  The scientific evidence for efficacy

 2.1.  There have been a number of systematic reviews and meta-analyses in this field, which as the Committee states are the best sources of evidence.   The most recent review of substance is that by Shang et al in 2005, which it considered “the most comprehensive to date” and which compared 110 placebo-controlled trials of hom­oeo­pathy [authors’ spelling] with 110 trials of conventional medicine matched for disorder and type of outcome.   The Committee cited a conclusion by the authors [paragraph 69] that “when analyses were restricted to large trials of higher quality there was no convinc­ing evidence that homeopathy [sic] was superior to placebo”.   They did not also cite the authors’ interpretation which followed these findings in the Lancet summary, which stated: “When account was taken for these biases [com­mon to trials of both homoeopathy and conventional medicine], there was weak evidence for a specific effect of homoeo­path­ic remedies, but strong evidence for specific effects of convent­ion­al interventions.   This finding is compatible with the notion that the clinical ef­fects of homoeopathy are placebo effects.”

 2.2.  This was no endorsement of homeopathy.   But it was some way removed from the Committee’s conclusion in paragraph 70 of their report, “In our view, the system­atic reviews and meta-analyses conclusively demonstrate that homeopathic products perform no better than placebos.”   It also provides little support for that part of Pro­fessor Ernst’s evidence to the Committee where he “pointed out that: . . . Shang et al very clearly arrived at a devastatingly negative overall conclusion” [67].

 2.3.  The exaggeration by the Committee of Shang’s conclusions is worrying.   It is difficult to see how a weakly supported positive effect, for which one explanation (poss­ibly well-found­ed) is a placebo effect, can be translated into a con­clus­ive demon­­stration of this effect, with a “devastatingly” negative finding.   No such firm claims can be found in Shang, who writes of finding “no strong” evidence, or “little” evidence, and who ends his paper with cautions about methodology and about the difficulty of detecting bias in studies, as well as the role of possible “con­text effects” in homeopathy.

 2.4.  The Committee’s overstatement is not helped by claiming Government sup­port for its interpretation in paragraph 70, based on the Minister’s concession of no “cred­ible” evidence that homeopathy works beyond placebo.   If he meant persuasive evid­ence  –  and his guarded support for further research [75] supports this  –  that shows a confusion by the Committee between absence of evidence and evidence of ab­sence.   If how­ever he was saying that all evidence was negative, this as Prof. Harper correctly stated [71] runs counter to the message from most reviews up to and including Shang, which is one of prim­ary studies of insufficient quantity, rigour, size, homo­­­­geneity and power to give clear-cut answers.

 2.5.  It is the absence of reliable evid­ence that re­mains the problem, and this includes evidence of an absence of spec­ific effects (while acknowledging the problem in prov­ing a negative, an obstacle which did not deflect the Committee from its conclus­ive verd­ict in 70).   The Commit­tee itself writes in 69 of no “convincing” evidence from Shang, from higher-quality trials, which is not consistent with a claim of con­clus­ive (dis)proof.   Care with words can be as import­ant as with figures, and can just as easily mislead.

 2.6.  In a search for best evidence in the early 2000s this author relied on the bullet­in on homeopathy produced by the NHS CRD at York in 2002, one of an Effect­ive Health Care series on “the effectiveness of health service interventions for decis­ion makers”.   This bulletin made a systematic assessment of the evidence to date.   It advised “caution” in interpreting this evidence, and warned that many of the areas researched were “not representative of the conditions that homeopathic practitioners usually treat”, and that “the methodological problems of the research” should be con­­­sidered.   It found “insufficient evidence of effectiveness . . to recommend homeo­pathy for any specific condition”.   At the same time it could not conclude that homeo­­­pathy performed no better than placebo.

 2.7.  That was eight years ago.   But it is notable that the more recent review by Shang, on which the Committee relied quite heavily, cited no reference to any placebo-controlled trial (i.e. of reasonable quality) subsequent to the CRD’s bulletin, in arriving at a suggestion, but not a conclusion, of a placebo effect.   The House of Commons Committee’s verdict in 70 stands on its own in going beyond what either review found from the evidence before it.

 2.8.  In seeking an up-to-date assessment from the NHS CRD, this author was re­ferred to the German researcher Klaus Linde as among the best of the objective sources of current evidence on homeopathy.   Linde, who was the lead author of a major review in 1997 cited by the Committee, in turn recommended the statist­ician Rainer Lüdtke as an expert with a good overview of the current literature.   Corres­pond­­ence ensued with both researchers, who were aware of the Committee’s recent report.

2.9.  Both Linde and Lüdtke hold that the Committee’s conclusion in 70 that reviews “conclusively demonstrate” a placebo effect is overstated and unsustainable on pre­sent evidence.   They have further criticisms of the way in which evidence has been addressed.

 2.10.  Both are critical of Prof. Ernst’s evidence to the Committee as highlighted in 67.   Prof. Linde confirms that his own 1999 re-analysis weakened the findings of his 1997 review and probably “at least overestimated the effects of homeopathic treat­ments”, but that his paper was “not ‘negative’” as stated by Ernst.   He writes that “A more accurate interpretation is that the ‘re-analyses’ [by himself and 5 others, referred to by Ernst] show that the (positive) evidence is not fool-proof.   This applies still today (for example, to the Shang analysis)”.   Lüdtke draws attention to his own paper in 2002 which criticised many statistical errors in Ernst’s 2000 re-analysis in the same journal, vitiating Ernst’s negative conclusion, a pub­lished criticism which received no mention in Ernst’s own evidence to the Commit­tee.   Ernst was correct to state in evidence elsewhere that the re-analyses of Linde came to a “less than positive” con­clusion, and that further reviews “failed to con­clude that homeopathy is effective”.   The Committee, while ad­opting Ernst’s more absolute conclusions, has not resolved the contra­diction between his statements.

 2.11.  Lüdtke, like Shang, has also drawn attention to the pitfalls in research into homeopathy, in a chapter in ‘New directions in homeopathy research’ (Witt C, Albrecht H, eds.) published in 2009.   He counsels against including all types of homeopathy trials of reason­able quality in one review (such reviews tend to suggest that homeo­pathic medicines are not efficacious), since the pooling of so many differ­ent kinds of trial and type of homeopathy makes findings unreli­able.   He advocates restricting systematic reviews to clearly defined health condit­ions or to single homeo­­pathic medicines, concluding that “the heterogeneity of trials is high and the meta-analysis results are not robust against small changes in study design or stat­istical analysis”.   In a paper published in 2008 he has argued that Shang’s conclus­ions do not hold when slightly different selection criteria are applied, e.g. by rede­fining how large is a “large” study, or by including treatment trials but excluding prevention trials.   Size is not the only factor in arriving at robust conclusions.

 2.12.  Context effects may play a part, according to both Shang and Lüdtke.   Shang’s “powerful alliances” between patient and carer, based on “shared strong beliefs”, may not be as distinctive or as peculiar to homeopathy as the nature of the homeo­pathic consultation, with its wider range of questions than are addressed in a con­ventional context, and the lifestyle recommendations referred to by Lüdtke that often flow from it.   There is overlap here with the placebo effect (see 4 below);  but homeo­pathy as “a complex medical system of its own” may be responsible for some broad­er effects.

 2.13.  Linde writes that the “undecided fraction” to which he belongs is confused by “the notorious lack of predictable reproducibility” on the one side, and by “too many anomalous results in high quality studies to rule out a relevant pheno­menon” on the other.

 3.  Other evidential considerations

 3.1  A conventional argument against CAM treatments is often that they are risky because they deny or delay a proper diagnosis and the adoption of tried and tested convent­ional treatments [105; 108; Ev 26-27].   But this is not an argument about (as here) homeopathy per se, and its side-effects which at such high dilutions are as im­plaus­ible as its efficacy is claimed to be.   The potential for harm however is real enough:  but only if patients have not been in contact with their own doct­ors, which happens in a minority of cases;  if homeopaths are not adequ­ately trained to recogn­ise ‘red flags’, and give bad advice;  and if conventional treat­ment is likely to be successful and/or ac­cept­ably risk-free in the particular case, and indeed more successful than a homeo­pathic approach.

 3.2.  The argu­ment for adopting one kind of treatment and not the other relates therefore to issues of practice, communication and training as well as of comparative effic­acy (for patient choice see 6.1 below).   These are highly important;  but it is not legit­imate to deploy the argument as the Committee did as a factor in the intrins­ic risk/benefit ratio of a therapy, which it is not, adducing it as an addit­ion­al negative ele­ment instead of as part of an efficacy argument which has already been addressed.   (Suppose high-quality trials establish homeopathy’s superiority over conventional treatment for a condition:  this, with homeopathy’s negligible side-effects, would make the conventional option the risky one.)

 3.3.  Nor is the argument even-handed if examination of true side-effects in homeo­pathic and conventional treatment is not ad­dressed when discussing the comparative merits of the two approaches, patient satis­fact­ion, and government policy.   Shang et al gave “the exclusive focus on benefic­ial ef­fects” as one of several limitat­ions of their study.   The extent of adverse clinical effects is as much a part of the evidence base as is benefit.   If the Committee had looked at these it might have cast a different light on policy towards homeo­pathy in the NHS, and would almost certain­ly have high­lighted public dis­quiet about some of the more aggressive conventional treat­ments as a reason for many patients preferring a CAM approach.   This is a significant omiss­ion.

 3.4.   There may be no good conventional treatment for a condition.   Alternatively, the standard treatment may be contraindicated.   The Committee has not considered these reasons why some patients may welcome the continued provision of homeo­pathy.

 4.  The placebo effect

 4.1.  The placebo effect, addressed at some length by the Committee (30–40), is not in dispute.   Yet much about it is unknown.   It may be premature to assume that pati­ent expectations of mod­ern medicine, with its erudit­ion, structures, scientific ap­proach and rituals which give it the intellectual and moral high ground in Western society, are of lesser force than those of a treatment which is commonly thought of as “implaus­ible”, and not only by scientists.   Belief in white coats is not weak.   Further­more patients are likely to resort to CAM on grounds of principle or safety as well as efficacy.   The placebo as an explanation is some­times reached for too readily off the shelf, when its applic­ab­il­ity to the relevant con­dition, treatment and patient popul­at­ion is poorly understood.   This gap in argu­ment has not been closed by the Commit­tee.   The placebo effect in homeopathy needs more work before conclusions can be confidently drawn.

 4.2.  Empathy in a consultation is more than a matter of time given [81]:  it also in­volves personality and training.   This author has on occasion felt better heard in a ten-minute GP consultation than in an hour with a CAM therapist, although the latter have generally shown up well.   The better comparator in CAM situations is probably the specialist consultation, since most patients will have initially visited their GPs.   Nor is it always the fluctuating or self-limiting conditions [43, 81], as the Committee suggests, that send patients to unconventional providers;  claimed relief from chronic complaints after a long period of failure with convention­al treatment is not un­common.

 4.3.  The surveys of homeopathic patients referred to in 80 suggest that self-reported benefit was not only at a high level but persisted beyond the limits of any placebo effect which, as the Committee states, is usually short-lived.

 5.  The Committee’s witnesses

 5.1.  The Committee in two sessions called twelve witnesses to give oral evidence, all but one with relevant affiliations.   Select­ion of witnesses can affect outcomes in the same way as selection of written evidence.   It is therefore legitimate to exam­ine the choices made.

5.2.  It is not easy to see why a journalist doctor was invited to appear in preference to some other non-representative con­tributors to the inquiry.   The written submiss­ion by Dr. Goldacre [Ev. 8] was not­ably short on support­ing evidence, but contained unqualified state­ments on the ineffect­ive­ness of homeo­pathy, forcefully expressed (“extreme quack­ery” was ment­ion­ed).   By contrast, the submission by the Comple­mentary Medic­ine Research Group from the Depart­ment of Health Sciences at the University of York presented a well-argued summary with 68 references [Ev. 143].   In this appears the state­ment “To date there are eight syst­em­atic reviews that pro­vide evid­ence that the effects of homeopathy are beyond placebo when used as a treatment for [five child­hood con­ditions]”.   This claim from a mainstream academic centre, rec­ently rated joint first nationally for health services research in the latest Research Assessment Exercise, stands in stark contra­diction to Prof. Ernst’s referenc­ed claims, noted above, and to Dr. Goldacre’s unreferenced state­ments.  It would have been enlightening if the Committee had probed the Group about this, face to face as a witness, and at­tempted some resolution before agreeing in unequivoc­al terms with the two witness­es who were invited to appear and were quoted favour­ably.

 The Committee critic­ised the supporters of homeopathy for their “selective ap­proach­­es” to evid­ence [73].   They could fairly be accused of the same.   Unfortunat­ely they did not (presumably) have the scope to solicit the views of Dr. Linde from Germany, which would have differed from those of Prof. Ernst with regard to the evidence.

 5.3.  Only one Primary Care Trust submitted a paper, and it was invited to give oral evidence on its decision that homeopathy did not provide value for money.   Given the number of PCTs countrywide this is rather surprising.   It might be wondered if some form of publication bias was in play, with the many PCTs who were happy with provision of homeopathy seeing no need to defend the status quo.   At least one writer complained of the short timescale for submissions [Ev. 128].   It would have been interesting to know what steps the Committee took to obtain a range of views about the evidence, and whether West Kent was the only PCT to have done an assessment of the kind referred to in Ev. 134.   Only a negative PCT view was re­corded;  and despite the Committee’s unequivocal conclusion even West Kent conceded “limited evidence in favour of homeopathy”.

 6.  Societal questions

 6.1.  Since doctors exist for patients and not the other way round it is not self-evident that scientific evidence, important as it is, should be the sole determinant of what is provided to the public.   If the patient is ult­im­ately in the driving seat (s)he might wish on broader grounds than proven efficacy to finance this type of treatment rather than that (or in addition to that) from the public purse.   This gives scope for political judgements which can set a government at odds with its medical advisers.   This should be no surprise to a parliamentary scient­ific committee which sits at the border of these two worlds.

 6.2.  In the purely scientific field it is interesting that the present Committee should feel “troubled” [71] by two senior government scientists coming to different conclus­ions about the weight of homeopathic evidence.   Such disagreement in interpretat­ion is quite common in scientific de­bate, although life is undoubtedly easier where there is con­sensus.   Premature con­sensus, however, has its own dangers, as is gener­ally recogn­ised.   The Commit­tee appears to require the scientists metaphor­ically to retire to a jury room and not come out until they agree [64, 72], presumably with the Com­mittee’s view.   This seems a step too far.

 6.3.  Pre-existing structures have some de facto­ claims.   It is reas­onable to decide that if something were not in existence one would not call it into being, but if it is already there one would not abolish it.   While theoreticians might debate this, society as a whole can accept it.   It is more easy to accept where the institution claims a minus­cule proportion of the health and research budgets, which must be the case with homeo­pathy whatever precise figure the government arrives at at the Committee’s request.

 7.  Conclusion

 7.1.  The evidence for homeopathy is not impressive, except possibly in terms of lack of adverse effects.   The Committee however has been less than rigorous in its ap­proach to this evidence.   Its choice of witnesses favoured a medical media opponent of homeopathy over a research centre of excellence.   It was unwise to rely heavily on the interpretations of one professor of CAM, some of whose statements are unsound or in conflict with other statements of his, and who is not without his critics in the worlds of research and academia whose views were given less prominence.   The 2005 review by Shang et al has been inaccurately represented as ruling out­ spec­if­ic effects of homeopathy, in a summary statement by the Committee that goes be­yond present evidence.   The Committee’s own statements show confusion between un­con­vincing evidence of a specific effect and disproof of it.   The true risk profile of homeopathy, compared with conventional treatment, was not considered.

7.2.  These limitations make the Committee’s report an unreliable source of evidence about homeopathy.   The jury must still be regarded as out on its efficacy and risk/ benefit ratio.   Whether more research should be done, and of what kind, is another question.   But there can be no ethical objection to it since the principal questions have not, as the Committee claimed, “been settled already” [78].

 Edward Baldwin.  June 2010.

Earl Baldwin of Bewdley was a member of the House of Lords’ Science and Technology Committee when they did a review of CAM in 2000. His critique questions the methodology used in the report, examining the selective uses and limitations of clinical evidence, and also critiques the scantiness of other evidential considerations.