Accuracy of one or two simple questions to identify alcohol-use disorder in primary care: a meta-analysis

Recently, in a meta-analysis conducted by Mitchell and colleagues, it was reported that one or two questions may have value in identifying alcohol misuse in primary care.1 Whilst this may be the case there are number of methodological concerns with this paper that potentially undermine some of its conclusions, and highlight some of the difficulties in conducting and interpreting diagnostic test accuracy meta-analyses.

One of the main concerns is the aggregating of different tests under the same heading. In this review there were 15 'studies' of the one- question (SQ) test for diagnosing alcohol-misuse but all the tests ('questions asked') were different. Similarly the two analyses of the two- question questionnaire (2QQ) analysed different tests. Critically the sensitivity and specificity varied between studies and therefore tests. This suggests that each unique question should be considered as a different test, and it has limited meaning to use meta-analysis to aggregate under the heading of one-question or two-question accuracy for identifying alcohol-misuse - it would be equivalent to aggregating ultrasound studies with MRI studies for diagnosing abdominal aortic aneurysms (AAA) under the heading of accuracy of imaging tests for diagnosing AAA.

Since different tests are being aggregated it is unsurprising that there is significant heterogeneity; inspection of the ROC plots show the sensitivity ranging from around 21% to nearly 90% and specificity from 50% to 97% between studies. It should also be noted that the I2 statistic is not very useful in bivariate meta-analyses. It is a univariate statistic and as a result is not recommended by the Cochrane Diagnostic Test Accuracy Working group.2 Nonetheless there is undoubted heterogeneity from inspection of the ROC plots.

To accommodate heterogeneity between studies, reviewers often appeal to a random effects model and this was the case here. Whilst it is true that the bivariate random effects model accommodates individual study parameters for the sensitivity and specificity, thus allowing these to vary between studies, this does not negate the fact that the summary sensitivity and specificity estimated by the model merely represent an average across all the studies. In the face of widespread heterogeneity, this average is unlikely to be representative of any particular setting or test, particularly when the latter varies between studies.

For the analysis of the 2QQ test there were only two studies aggregated. It is at best speculative to draw strong conclusions when there are so few studies to meta-analyse. Indeed, the STATA package Metandi, used for conducting bivariate random effects meta-analysis and used in this study, will not aggregate studies when there are fewer than four for this reason.

The quoted prevalence of 21% also shows widespread variation (12-41%). This may be due to differences in disease definition (the reference standard was not consistent across analyses), as well as differences in patient population, such as spectrum effects.3 Thus, it is unclear which population this average represents.

Furthermore when considering the combined effect of the SQ test with say AUDIT, it is often implicitly assumed (as it is here) that the two tests are independent, and retain the same performance characteristics (sensitivity/specificity) irrespective of the prevalence. Independence could potentially be violated if all or part of the SQ test is incorporated implicitly or explicitly within AUDIT - as may be the case with some of the SQ tests. Also the prevalence has been shown to affect the sensitivity and specificity of diagnostics tests, particularly with tests that have an implicit threshold such as the ones used here,4,5 thus threatening the assumption that performance is not affected by prevalence.

This does raise the question of whether there is any validity to the results of such analyses?

From the 'real-world' analyses, the authors state that for '100 hypothetical attendee(s) at a prevalence of 21%' the SQ would identify 11 correctly (missing ten), correctly rule out 69 and falsely diagnose 10. Unfortunately these figures have no or limited potential for transferability to practice since it is unclear what we mean by the average SQ test, especially when the sensitivity and specificity of the SQ test depends critically on which questions are being used, and possibly on the prevalence.

In addition, similar claims on the number of true/false positive/negatives for the combined algorithm are undermined by the above observations, and the likely conditional dependence between the combined tests.

It is an unfortunate fact with a lot of systematic reviews and meta- analyses that we are often unable to draw strong conclusions, particularly when there are few studies or significant heterogeneity. The latter especially affects the applicability of results in practice and it is only recently that methods have been developed to deal with this.6

The question of whether, in general, GPs should use SQ/2QQs in combination with other questionnaire tools is not answered by this study as it seems to depend significantly on both the type of SQ/2QQ used and the target population. Furthermore, given none of the SQ/2QQ's was evaluated more than once, even the most promising examples would need greater investigation before recommending their implementation in practice.

References

1. Mitchell AJ, Bird V, Rizzo M, Hussain S, Meader N. Accuracy of one or two simple questions to identify alcohol-use disorder in primary care: a meta-analysis. Br J Gen Pract 2014; DOI: 10.3399/bjgp14X680497

2. Bossuyt P, Davenport C, Deeks J, Hyde C, Leeflang M, Scholten R. Chapter 11: Interpreting results and drawing conclusions. In: Deeks JJ, Bossuyt PM, Gatsonis C (editors), Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy Version 0.9. The Cochrane Collaboration, 2013. Available from: http://srdta.cochrane.org/.

3. Willis BH. Spectrum bias--why clinicians need to be cautious when applying diagnostic test studies. Fam Pract 2008; 25:390-396.

4. Willis BH. Evidence that disease prevalence may affect the performance of diagnostic tests with an implicit threshold: a cross sectional study. BMJ Open 2012; 2:e000746

5. Leeflang MM, Rutjes AW, Reitsma JB, Hooft L, Bossuyt PM. Variation of a test's sensitivity and specificity with disease prevalence. CMAJ. 2013; 185(11):E537-44. doi: 10.1503/cmaj.121286

6. Willis BH, Hyde CJ. Estimating a test's accuracy using tailored meta-analysis - How setting-specific data may aid study selection. J Clin Epidemiol 2014; 67:538-546

Conflict of Interest:

None declared