The validity and reliability of an interactive computer tobacco and alcohol use survey in general practice

Background: Uncertainty regarding the accuracy of the computer as a data collection or patient screening tool persists. Previous research evaluating the validity of computer health surveys have tended to compare those responses to that of paper survey or clinical interview (as the gold standard). This approach is limited as it assumes that the paper version of the self-report survey is valid and an appropriate gold standard.

Objectives: First, to compare the accuracy of computer and paper methods of assessing self-reported smoking and alcohol use in general practice with biochemical measures as gold standard. Second, to compare the test re-test reliability of computer administration, paper administration and mixed methods of assessing self-reported smoking status and alcohol use in general practice.

Methods: A randomised cross-over design was used. Consenting patients were randomly assigned to one of four groups; Group 1. C-C : completing a computer survey at the time of that consultation (Time 1) and a computer survey 4-7 days later (Time 2); Group 2. C-P: completing a computer survey at Time 1 and a paper survey at Time 2; Group 3. P-C: completing a paper survey at Time 1 and a computer survey at Time 2; and Group 4. P-P: completing a paper survey at Time 1 and 2. At Time 1 all participants also completed biochemical measures to validate self-reported smoking status (expired air carbon monoxide breath test) and alcohol consumption (ethyl alcohol urine assay).

Results: Of the 618 who were eligible, 575 (93%) consented to completing the Time 1 surveys. Of these, 71% (N=411) completed Time 2 surveys. Compared to CO, the computer smoking self-report survey demonstrated 91% sensitivity, 94% specificity, 75% positive predictive value (PPV) and 98% negative predictive value (NPV). The equivalent paper survey demonstrated 86% sensitivity, 95% specificity, 80% PPV, and 96% NPV. Compared to urine assay, the computer alcohol use self-report survey demonstrated 92% sensitivity, 50% specificity, 10% PPV and 99% NPV. The equivalent paper survey demonstrated 75% sensitivity, 57% specificity, 6% PPV, and 98% NPV. Level of agreement of smoking self-reports at Time 1 and Time 2 revealed kappa coefficients ranging from 0.95 to 0.98 in each group and hazardous alcohol use self-reports at Time 1 and Time 2 revealed kappa coefficients ranging from 0.90 to 0.96 in each group.

Conclusion: The collection of self-reported health risk information is equally accurate and reliable using computer interface in the general practice setting as traditional paper survey. Computer survey appears highly reliable and accurate for the measurement of smoking status. Further research is needed to confirm the adequacy of the quantity/frequency measure in detecting those who drink alcohol. Interactive computer administered health surveys offer a number of advantages to researchers and clinicians and further research is warranted.