Elsevier

Addictive Behaviors

Volume 35, Issue 5, May 2010, Pages 492-498
Addictive Behaviors

The validity and reliability of an interactive computer tobacco and alcohol use survey in general practice

https://doi.org/10.1016/j.addbeh.2009.12.030Get rights and content

Abstract

Background

Uncertainty regarding the accuracy of the computer as a data collection or patient screening tool persists. Previous research evaluating the validity of computer health surveys have tended to compare those responses to that of paper survey or clinical interview (as the gold standard). This approach is limited as it assumes that the paper version of the self-report survey is valid and an appropriate gold standard.

Objectives

First, to compare the accuracy of computer and paper methods of assessing self-reported smoking and alcohol use in general practice with biochemical measures as gold standard. Second, to compare the test re-test reliability of computer administration, paper administration and mixed methods of assessing self-reported smoking status and alcohol use in general practice.

Methods

A randomised cross-over design was used. Consenting patients were randomly assigned to one of four groups; Group 1. C–C : completing a computer survey at the time of that consultation (Time 1) and a computer survey 4–7 days later (Time 2); Group 2. C–P: completing a computer survey at Time 1 and a paper survey at Time 2; Group 3. P–C: completing a paper survey at Time 1 and a computer survey at Time 2; and Group 4. P–P: completing a paper survey at Time 1 and 2. At Time 1 all participants also completed biochemical measures to validate self-reported smoking status (expired air carbon monoxide breath test) and alcohol consumption (ethyl alcohol urine assay).

Results

Of the 618 who were eligible, 575 (93%) consented to completing the Time 1 surveys. Of these, 71% (N = 411) completed Time 2 surveys. Compared to CO, the computer smoking self-report survey demonstrated 91% sensitivity, 94% specificity, 75% positive predictive value (PPV) and 98% negative predictive value (NPV). The equivalent paper survey demonstrated 86% sensitivity, 95% specificity, 80% PPV, and 96% NPV. Compared to urine assay, the computer alcohol use self-report survey demonstrated 92% sensitivity, 50% specificity, 10% PPV and 99% NPV. The equivalent paper survey demonstrated 75% sensitivity, 57% specificity, 6% PPV, and 98% NPV. Level of agreement of smoking self-reports at Time 1 and Time 2 revealed kappa coefficients ranging from 0.95 to 0.98 in each group and hazardous alcohol use self-reports at Time 1 and Time 2 revealed kappa coefficients ranging from 0.90 to 0.96 in each group.

Conclusion

The collection of self-reported health risk information is equally accurate and reliable using computer interface in the general practice setting as traditional paper survey. Computer survey appears highly reliable and accurate for the measurement of smoking status. Further research is needed to confirm the adequacy of the quantity/frequency measure in detecting those who drink alcohol. Interactive computer administered health surveys offer a number of advantages to researchers and clinicians and further research is warranted.

Introduction

The use of interactive computers to collect health data directly from patients has several advantages (Fricker & Schonlau, 2002). Studies indicate that computers are a confidential, acceptable, feasible, user-preferred, and cost-effective mechanism for collecting health information (Bernhardt et al., 2001, Wright et al., 1998, Hibbert et al., 1996, Shakeshaft et al., 1998, Shakeshaft and Frankish, 2003), particularly if the touch screen format is used (Westman, Hampel & Bradley, 2000). Touch screen computer interface has reported high acceptability rates in settings such as general practice (Bonevski, Sanson-Fisher, Campbell & Ireland, 1997) community drug and alcohol clinics (Shakeshaft, Bowman & Sanson-Fisher, 1998), and cancer treatment centres (Newell, Girgis & Sanson-Fisher, 1997). Computerised surveys offer better flexibility in questionnaire design due to automatic tailoring and branching of items based on responses, reducing item redundancy and missing data (Fricker & Schonlau, 2002). An additional advantage of the computerised data collection is that it can provide tailored ‘real time’ results immediately available to users and their doctors (Bonevski, Sanson-Fisher, Campbell, Carruthers, Reid & Ireland, 1999). This quality in particular has motivated the proliferation of computer-delivered and web-based health behaviour change interventions (Portnoy et al., 2008, Walters et al., 2006).

The practicality and acceptability of computerised self-report measures cannot always compensate for other important characteristics. Uncertainty regarding the accuracy and reliability of the computer as a data collection or patient screening tool persists despite the importance of these characteristics (Fricker & Schonlau, 2002). Inaccurate self-report measures may lead to misclassification of patient's health risk status which may in turn result in the use of inappropriate interventions. Given many factors including mode of delivery (Bowling, 2005) may affect the accuracy of a self-report measure, it is imperative that new methods of collecting health-related self-report information are thoroughly evaluated. The accuracy of self-report measures can be assessed in several ways (Nunnally, 1978). One desirable quality that a measure should demonstrate is concurrent validity. Assessing concurrent validity usually involves the comparison of self-reported health behaviours to a gold standard (or true) measure of the behaviour. Another important characteristic is the demonstrated test retest reliability. Reliability is the extent to which a measure produces results which are free of random error and can be assessed by measuring the degree of discrepancy of responses when a scale is administered to a sample on two separate occasions.

Previous research evaluating the validity of computer-delivered health behaviour surveys have tended to compare the responses of the computer survey to that of paper survey or clinical interview (as the gold standard) (Ahmed et al., 2008, Wu et al., 2009, Hayward et al., 1992, Davis et al., 1992). This approach is limited as it assumes that the paper version of the self-report survey is valid and an appropriate gold standard. For example, there is some evidence that responses to computer surveys can be more valid than responses to face-to-face interviews, particularly for socially undesirable behaviours (Beck et al., 1988, Greist et al., 2000). Surprisingly few studies have compared commonly assessed health risk behaviours using computerised surveys against biochemical measures. One such study has compared self-report from 52 general practice patients using computer survey on diet and nutrition with blood and urine assay results as gold standard (O'Donnell, Nelson, Wise & Walker, 1991). The authors failed to provide any information on the appropriateness of this gold standard as a measure of dietary intake. Information on the sensitivity and specificity of the assays and the half-life of the biochemical variables in these mediums is necessary. The study also failed to provide a comparison of accuracy with non-computerised methods. Another study compared computer survey self-reported and accelerometer-measured average daily time spent performing moderate to vigorous physical activity (Wong, Leatherdale & Manske, 2006). Self-reported and measured BMI were also compared. The study found that computer collected self-report responses significantly correlated with the objective gold standard measures. However, important indices of agreement were not reported such as sensitivity and specificity of the self-report measure. The study also failed to compare the computer survey results with non-computerised alternative survey methods. Finally the generalisability of those results is restricted as the study was set in secondary schools with students in grades 6 to 12.

A number of studies have examined the test re-test reliability of computer delivery of health risk survey and have generally concluded that computer surveys are reliable (Hayward et al., 1992, Bernadt et al., 1989, Miller et al., 2002). However, many of these studies have had small sample sizes (Bernadt et al., 1989, Miller et al., 2002), or short periods (1–4 h) between the first and second administration of the survey (Hayward, Smittner, Meyer et al, 1992).

The aims of this study are two-fold. Firstly, to compare the accuracy of computer and paper methods of assessing self-reported smoking and alcohol use in general practice with biochemical measures as gold standard. Secondly, to compare the test re-test reliability of computer administration, paper administration and administration using mixed methods of assessing self-reported smoking status and alcohol use in general practice.

Section snippets

Study design

A randomised cross-over design was used (see Fig. 1). Consenting patients were randomly assigned to one of four groups;

Group 1.C–C: those completing a computer survey at the time of that consultation (Time 1) and a computer survey 4–7 days later (Time 2).
Group 2.C–P: those completing a computer survey at Time 1 and a paper survey at Time 2.
Group 3.P–C: those completing a paper survey at Time 1 and a computer survey at Time 2.
Group 4.P–P: those completing paper surveys at Time 1 and Time 2.

Recruitment

A

Sample

A total of 878 patients were approached by the interviewers. Of these, 260 (30%) were ineligible; 58 were under 18 years of age, 77 displayed limited or no English, 11 were too ill, and 114 had previously participated. Of the 618 who were eligible, 575 (93%) consented to completing the Time 1 surveys. In summary, of the participants who completed the Time 1 surveys, 95% (N = 549) completed the CO test and 87% (N = 499) completed the urinalysis and 71% (N = 411) completed Time 2 surveys. Fig. 1 shows

Discussion

This study aimed to examine the psychometric qualities of interactive computer administered patient health risk surveys in the general practice setting and compare those qualities with traditional paper administration of surveys. The study found no notable differences between administration modes in prevalence of estimates of smoking and alcohol use as indicated by overlapping 95% confidence intervals. The results are in accord with previous studies, but provide new information on the

Conclusion

In conclusion, the collection of self-reported health risk information is equally accurate and reliable using computer administered survey in the general practice setting as paper administered survey. Computer health risk assessment appears highly reliable and accurate for the measurement of smoking status. Further research is needed to confirm the adequacy of the quantity/frequency measure in detecting those who drink alcohol. Whilst the sensitivity and test re-test reliability of the computer

Role of Funding Source

This research was funded by the Commonwealth General Practice Evaluation Program and Cancer Council New South Wales' Centre for Health Research & Psycho-oncology (CHeRP). The funders played no role in the conduct of the study.

Contributors

All authors contributed to the conceptual and planning phases, data collection, data analysis and writing of the paper.

Conflict of Interest

The authors declare no known conflicts of interest.

Acknowledgements

This research was funded by the Commonwealth General Practice Evaluation Program and Cancer Council New South Wales' Centre for Health Research & Psycho-oncology (CHeRP). The views expressed are not necessarily those of The Cancer Council.

We gratefully acknowledge the assistance of Professor ALA Reid and Professor M. Ireland in the recruitment of general practitioners and Ms L. Adamson in the recruitment of patients.

References (42)

  • M.W. Bernadt et al.

    Can a computer reliably elicit an alcohol history?

    British Journal of Addiction

    (1989)
  • J.M. Bernhardt et al.

    Handheld computer-assisted self interviews: user comfort level and preferences

    American Journal of Health Behaviour

    (2001)
  • B. Bonevski et al.

    Do general practice patients find computer health risk surveys acceptable? A comparison with pen-and-paper method

    Health Promotion Journal of Australia

    (1997)
  • A. Bowling

    Mode of questionnaire administration can have serious effects on data quality

    Journal of Public Health

    (2005)
  • K.M. Conigrave et al.

    Diagnostic tests for alcohol consumption

    Alcohol and Alcoholism

    (1995)
  • L.J. Davis et al.

    Substance use disorder diagnostic schedule (SUDDS): the equivalence and validity of a computer-administered and an interviewer-administered format

    Alcoholism, Clinical and Experimental Research

    (1992)
  • S. Dawe et al.
  • J.A. Dickinson et al.

    General practitioners' detection of patients' smoking status

    Medical Journal of Australia

    (1989)
  • R.D. Fricker et al.

    Advantages and disadvantages of internet research surveys: evidence from the literature

    Field Methods

    (2002)
  • J.H. Greist et al.

    Computer interview questionnaires for drug use/abuse

  • R.S.A. Hayward et al.

    Computer versus interview administered preventive care questionnaire: does survey medium affect response reliability?

    Clinical Research

    (1992)
  • Cited by (21)

    • Effect of electronic screening and brief intervention on hazardous or harmful drinking among adults in the hospital outpatient setting: A randomized, double-blind, controlled trial

      2018, Drug and Alcohol Dependence
      Citation Excerpt :

      The main limitation of our study, which is typical of the existing literature, was the self-report of co-primary outcomes. Although blood markers (for example, gamma-glutamyltransferase level) may seem preferable for assessing outcomes, they have low sensitivity to hazardous drinking, and self-report has generally been found to be reliable (Kypri et al., 2015), particularly via computers (Bonevski et al., 2010; Miller et al., 2002). Strengths of the study include allocation concealment from research staff, blinding of participants to the study design (they were not informed it was a trial, as ethically approved, until after the 12-month follow-up Johnson et al., 2013), and a satisfactory rate of follow-up for primary outcomes.

    • Prevalence of unhealthy alcohol use in hospital outpatients

      2014, Drug and Alcohol Dependence
      Citation Excerpt :

      Strengths of the study include the use of a valid screening instrument (Bradley et al., 2007), a pre-tested study procedure (Johnson et al., 2013a), and the large sample size. Although self-reported alcohol consumption via computers has been shown to be reliable (Bonevski et al., 2010; Lotfipour et al., 2012), the main limitation of this study is the reliance on self-report, especially since many participants completed the AUDIT-C in close proximity to other people. Another limitation was the non-collection of alcohol consumption data from 40% of the adult hospital outpatients we invited.

    • Smoking among dental students at King Saud University: Consumption patterns and risk factors

      2014, Saudi Dental Journal
      Citation Excerpt :

      One limitation of this study is related to the issue of validity for self-reported information, a common issue for surveys. Self-reported smoking status can agree with biomarker tests for tobacco use (Wong et al., 2012; Bonevski et al., 2010; Chen et al., 2002), but in some circumstances, such as with teenagers or pregnant women, self-reported smoking data may not be valid (Caraballo et al., 2004; Fakhfakh et al., 2011). Anonymous surveys or online surveys similar to the one used in this study generally have high reliability and validity (Ramo et al., 2011; Klein et al., 2007).

    • Risk factors for intracerebral hemorrhage differ according to hemorrhage location

      2012, Neurology
      Citation Excerpt :

      Furthermore, self-report does not account for the severity of the hypertension or hypercholesterolemia. Sensitivities for self-report of hypertension, hypercholesterolemia, tobacco use, and alcohol consumption in prior studies ranged from 51%–86%, and specificities ranged from 57%–95%.36–38 Cases and controls had risk factor assessment by the same methodology, however, and common atherosclerotic risk factors such as diabetes and cigarette smoking did not show significant differences between cases and controls, which suggests that recall bias was not a significant factor in our analysis.

    View all citing articles on Scopus
    View full text