The report of the CACTUS study1 and the accompanying editorial2 are flawed by several biases and errors. The Editor's review gave the study an unjustified commendation by stating, ‘A series of five-element acupuncture treatments has significant and sustained benefit in patients who frequently attend with medically unexplained symptoms’.3 In fact, the flaws and biases are so many that we can expose only the most important ones: failure to consider clinical relevance and measurement precision, and failure to consider the risk of bias in unblinded pragmatic trials such as CACTUS.
The differences in outcome measures are small and imprecise and therefore unlikely to be relevant to patients. For example, the primary outcome measure in CACTUS was the Measure Yourself Medical Outcome Profile score at 26 weeks. For this outcome the difference was only -0.6 on a 7 point scale with a wide 95% confidence interval (-1.1 to 0.0). Had the graphs in Figure 2 shown the confidence intervals rather than the point estimates alone, it would have been inescapably obvious that, although some results are statistically significant, no results are clinically important.
When differences are statistically significant, the results cannot easily be explained by chance. However, this does not mean that they cannot easily be explained by bias, and the onus is on authors to justify assumptions that the risk of bias is small. Like many other advocates of acupuncture and integrative medicine, the authors and editorialists fail to understand two important limitations of unblinded pragmatic trials that rely on subjective outcome measures: first, that their results have a high potential for bias, and second, that the size of the bias effect is likely to be larger than effects specific to acupuncture.4
The purpose of having a control group to control for biases is undermined if there is no blinding and the control is not a true control, that is to say, similar in all aspects to the treatment group except for the treatment. In the CACTUS study, control group participants were not similar for, as the authors acknowledge, they were likely to have been unhappy about not receiving acupuncture. They were likely, therefore, to experience a negative placebo reaction, or as we have termed it, a ‘frustrebo reaction’.5 The measured placebo effect will be the sum of the negative placebo or ‘frustrebo effect’ in the control group plus the positive placebo effect in the treatment group. To the unwary, the harm to the control group will appear as a benefit to the treatment group and the overall benefit of treatment (if any) will be exaggerated. We have explained this in greater detail elsewhere.5 The statement that the statistician was blinded is irrelevant and serves no purpose other than to suggest some degree of objectivity.
It is a pity that the opportunity provided by the qualitative study to investigate likely causes of a negative placebo effect was lost. By restricting the qualitative study to those who had completed acupuncture, the chance was missed to capture the experience of those who had been denied it. A comparison between the test and control groups' feelings at 26 weeks would have been very informative.
It is convenient for advocates of complementary and alternative medicine that clinical relevance and the role of bias have been overlooked once again.
- © British Journal of General Practice 2011
