We have all heard of the ‘hierarchy of evidence’. It describes a hierarchy of study designs for testing the effectiveness of therapeutic interventions and enables us to contemplate the relative merits of different types of investigations. In my field, complementary medicine, the logic behind this hierarchy has remained a hotly disputed topic.1 Many believers in complementary medicine seem to reject it and some even seem to have started promoting something I call the ‘anarchy of evidence’.
Enthusiasts of this or that complementary therapy invariably seem to be in favour of evidence-based medicine — but only as long as its application to their subject generates the results they had hoped for! Whenever the evidence fails to show that their therapy is effective, they call for a different standard. The reason is simple: enthusiasts are led by belief rather than evidence: if a rigorous randomised clinical trial does not demonstrate that their therapy is effective, it usually is not the treatment but the test that is deemed to be at fault. The thought that their belief was wrong is unthinkable to believers.
An example of this concept comes from the recent report by the Kings' Fund Assessing Complementary Practice: Building Consensus on Appropriate Research Methods.2 The argument strongly promoted in this report is that:
‘… discounting of the placebo-related aspects of an intervention may reduce its value …’.2
The authors of the report therefore believe, we should use different research tools:
'… the test can become one of ‘usual treatment’ against ‘usual treatment plus complementary practice.’2
Pragmatic studies can, of course, be very useful — but are primarily for testing how well a treatment performs in real life, once rigorous studies have demonstrated it to be effective under well-controlled conditions. Pragmatic studies deliberately do not control for placebo and other non-specific effects. We have recently published a systematic review of such trials in the area of acupuncture.3 Our results indicate that the probability of such a study design ever producing a negative result approaches zero. The non-specific effects of the tested intervention will almost invariably generate a positive result, if subjective outcome measures such as pain, wellbeing, or quality of life are employed, and if the study is large enough and employs sufficiently sensitive outcome measures. Even if the intervention is entirely devoid of specific effects, such a study will generate a (false) positive result. In other words, this test can be passed with flying colours even by pure placebos.
But the proponents of the anarchy of evidence go one decisive step further. If a treatment should not even pass the test of a pragmatic trial — those therapies which fail to generate powerful placebo effects might belong to this category — the standard must be lowered further. The general aim, of course, is to avoid the embarrassment of a negative result. Some complementary therapists already argue that observational studies without a control group might provide valuable data about the effectiveness of their intervention.4 Whenever disappointing results from pragmatic trials emerge, this approach can be used to turn the negative into a (false) positive. It is able to generate positive outcomes purely on the strength of the natural history of diseases and the regression towards the mean.
The anarchy of evidence has a downside, of course, it will mislead healthcare professionals and hinder progress. But, in the eyes of the enthusiasts of complementary medicine, this might well be a price worth paying.
- © British Journal of General Practice, 2010.
