The validity and reliability of an interactive computer tobacco and alcohol use survey in general practice
Introduction
The use of interactive computers to collect health data directly from patients has several advantages (Fricker & Schonlau, 2002). Studies indicate that computers are a confidential, acceptable, feasible, user-preferred, and cost-effective mechanism for collecting health information (Bernhardt et al., 2001, Wright et al., 1998, Hibbert et al., 1996, Shakeshaft et al., 1998, Shakeshaft and Frankish, 2003), particularly if the touch screen format is used (Westman, Hampel & Bradley, 2000). Touch screen computer interface has reported high acceptability rates in settings such as general practice (Bonevski, Sanson-Fisher, Campbell & Ireland, 1997) community drug and alcohol clinics (Shakeshaft, Bowman & Sanson-Fisher, 1998), and cancer treatment centres (Newell, Girgis & Sanson-Fisher, 1997). Computerised surveys offer better flexibility in questionnaire design due to automatic tailoring and branching of items based on responses, reducing item redundancy and missing data (Fricker & Schonlau, 2002). An additional advantage of the computerised data collection is that it can provide tailored ‘real time’ results immediately available to users and their doctors (Bonevski, Sanson-Fisher, Campbell, Carruthers, Reid & Ireland, 1999). This quality in particular has motivated the proliferation of computer-delivered and web-based health behaviour change interventions (Portnoy et al., 2008, Walters et al., 2006).
The practicality and acceptability of computerised self-report measures cannot always compensate for other important characteristics. Uncertainty regarding the accuracy and reliability of the computer as a data collection or patient screening tool persists despite the importance of these characteristics (Fricker & Schonlau, 2002). Inaccurate self-report measures may lead to misclassification of patient's health risk status which may in turn result in the use of inappropriate interventions. Given many factors including mode of delivery (Bowling, 2005) may affect the accuracy of a self-report measure, it is imperative that new methods of collecting health-related self-report information are thoroughly evaluated. The accuracy of self-report measures can be assessed in several ways (Nunnally, 1978). One desirable quality that a measure should demonstrate is concurrent validity. Assessing concurrent validity usually involves the comparison of self-reported health behaviours to a gold standard (or true) measure of the behaviour. Another important characteristic is the demonstrated test retest reliability. Reliability is the extent to which a measure produces results which are free of random error and can be assessed by measuring the degree of discrepancy of responses when a scale is administered to a sample on two separate occasions.
Previous research evaluating the validity of computer-delivered health behaviour surveys have tended to compare the responses of the computer survey to that of paper survey or clinical interview (as the gold standard) (Ahmed et al., 2008, Wu et al., 2009, Hayward et al., 1992, Davis et al., 1992). This approach is limited as it assumes that the paper version of the self-report survey is valid and an appropriate gold standard. For example, there is some evidence that responses to computer surveys can be more valid than responses to face-to-face interviews, particularly for socially undesirable behaviours (Beck et al., 1988, Greist et al., 2000). Surprisingly few studies have compared commonly assessed health risk behaviours using computerised surveys against biochemical measures. One such study has compared self-report from 52 general practice patients using computer survey on diet and nutrition with blood and urine assay results as gold standard (O'Donnell, Nelson, Wise & Walker, 1991). The authors failed to provide any information on the appropriateness of this gold standard as a measure of dietary intake. Information on the sensitivity and specificity of the assays and the half-life of the biochemical variables in these mediums is necessary. The study also failed to provide a comparison of accuracy with non-computerised methods. Another study compared computer survey self-reported and accelerometer-measured average daily time spent performing moderate to vigorous physical activity (Wong, Leatherdale & Manske, 2006). Self-reported and measured BMI were also compared. The study found that computer collected self-report responses significantly correlated with the objective gold standard measures. However, important indices of agreement were not reported such as sensitivity and specificity of the self-report measure. The study also failed to compare the computer survey results with non-computerised alternative survey methods. Finally the generalisability of those results is restricted as the study was set in secondary schools with students in grades 6 to 12.
A number of studies have examined the test re-test reliability of computer delivery of health risk survey and have generally concluded that computer surveys are reliable (Hayward et al., 1992, Bernadt et al., 1989, Miller et al., 2002). However, many of these studies have had small sample sizes (Bernadt et al., 1989, Miller et al., 2002), or short periods (1–4 h) between the first and second administration of the survey (Hayward, Smittner, Meyer et al, 1992).
The aims of this study are two-fold. Firstly, to compare the accuracy of computer and paper methods of assessing self-reported smoking and alcohol use in general practice with biochemical measures as gold standard. Secondly, to compare the test re-test reliability of computer administration, paper administration and administration using mixed methods of assessing self-reported smoking status and alcohol use in general practice.
Section snippets
Study design
A randomised cross-over design was used (see Fig. 1). Consenting patients were randomly assigned to one of four groups;Group 1. C–C: those completing a computer survey at the time of that consultation (Time 1) and a computer survey 4–7 days later (Time 2). Group 2. C–P: those completing a computer survey at Time 1 and a paper survey at Time 2. Group 3. P–C: those completing a paper survey at Time 1 and a computer survey at Time 2. Group 4. P–P: those completing paper surveys at Time 1 and Time 2.
Recruitment
A
Sample
A total of 878 patients were approached by the interviewers. Of these, 260 (30%) were ineligible; 58 were under 18 years of age, 77 displayed limited or no English, 11 were too ill, and 114 had previously participated. Of the 618 who were eligible, 575 (93%) consented to completing the Time 1 surveys. In summary, of the participants who completed the Time 1 surveys, 95% (N = 549) completed the CO test and 87% (N = 499) completed the urinalysis and 71% (N = 411) completed Time 2 surveys. Fig. 1 shows
Discussion
This study aimed to examine the psychometric qualities of interactive computer administered patient health risk surveys in the general practice setting and compare those qualities with traditional paper administration of surveys. The study found no notable differences between administration modes in prevalence of estimates of smoking and alcohol use as indicated by overlapping 95% confidence intervals. The results are in accord with previous studies, but provide new information on the
Conclusion
In conclusion, the collection of self-reported health risk information is equally accurate and reliable using computer administered survey in the general practice setting as paper administered survey. Computer health risk assessment appears highly reliable and accurate for the measurement of smoking status. Further research is needed to confirm the adequacy of the quantity/frequency measure in detecting those who drink alcohol. Whilst the sensitivity and test re-test reliability of the computer
Role of Funding Source
This research was funded by the Commonwealth General Practice Evaluation Program and Cancer Council New South Wales' Centre for Health Research & Psycho-oncology (CHeRP). The funders played no role in the conduct of the study.
Contributors
All authors contributed to the conceptual and planning phases, data collection, data analysis and writing of the paper.
Conflict of Interest
The authors declare no known conflicts of interest.
Acknowledgements
This research was funded by the Commonwealth General Practice Evaluation Program and Cancer Council New South Wales' Centre for Health Research & Psycho-oncology (CHeRP). The views expressed are not necessarily those of The Cancer Council.
We gratefully acknowledge the assistance of Professor ALA Reid and Professor M. Ireland in the recruitment of general practitioners and Ms L. Adamson in the recruitment of patients.
References (42)
- et al.
Randomized controlled trial of a computer strategy to increase general practitioner preventive care
Preventive Medicine
(1999) - et al.
Computer-delivered interventions for health promotion and behavioural risk reduction: a meta-analysis of 75 randomized controlled trials, 1988–2007
Preventive Medicine
(2008) - et al.
Computers in community-based drug and alcohol clinical settings: are they acceptable to respondents?
Drug & Alcohol Dependence
(1998) - et al.
A review of computer and internet-based interventions for smoking behaviour
Addictive Behaviours
(2006) - et al.
The reliability and stability of a quantity–frequency method and a diary method of measuring alcohol consumption
Drug and Alcohol Dependence
(1991) - et al.
Assessing patient attitudes to computerized screening in primary care: psychometric properties of the computerized lifestyle assessment scale
Journal Medical Internet Research
(2008) Methodological issues in self-report of health behaviour
Journal of School Health
(1985)- et al.
Computerised administration of alcoholism screening tests in a primary care setting
Journal of the American Board of Family Practice
(1990) - et al.
Scale for suicide ideation: psychometric properties of self report version
Journal of Clinical Psychology
(1988) for the SRNT Subcommittee on Biochemical Verification. Biochemical verification of tobacco use and cessation
Nicotine & Tobacco Research
(2002)
Can a computer reliably elicit an alcohol history?
British Journal of Addiction
Handheld computer-assisted self interviews: user comfort level and preferences
American Journal of Health Behaviour
Do general practice patients find computer health risk surveys acceptable? A comparison with pen-and-paper method
Health Promotion Journal of Australia
Mode of questionnaire administration can have serious effects on data quality
Journal of Public Health
Diagnostic tests for alcohol consumption
Alcohol and Alcoholism
Substance use disorder diagnostic schedule (SUDDS): the equivalence and validity of a computer-administered and an interviewer-administered format
Alcoholism, Clinical and Experimental Research
General practitioners' detection of patients' smoking status
Medical Journal of Australia
Advantages and disadvantages of internet research surveys: evidence from the literature
Field Methods
Computer interview questionnaires for drug use/abuse
Computer versus interview administered preventive care questionnaire: does survey medium affect response reliability?
Clinical Research
Cited by (21)
Effect of electronic screening and brief intervention on hazardous or harmful drinking among adults in the hospital outpatient setting: A randomized, double-blind, controlled trial
2018, Drug and Alcohol DependenceCitation Excerpt :The main limitation of our study, which is typical of the existing literature, was the self-report of co-primary outcomes. Although blood markers (for example, gamma-glutamyltransferase level) may seem preferable for assessing outcomes, they have low sensitivity to hazardous drinking, and self-report has generally been found to be reliable (Kypri et al., 2015), particularly via computers (Bonevski et al., 2010; Miller et al., 2002). Strengths of the study include allocation concealment from research staff, blinding of participants to the study design (they were not informed it was a trial, as ethically approved, until after the 12-month follow-up Johnson et al., 2013), and a satisfactory rate of follow-up for primary outcomes.
Prevalence of unhealthy alcohol use in hospital outpatients
2014, Drug and Alcohol DependenceCitation Excerpt :Strengths of the study include the use of a valid screening instrument (Bradley et al., 2007), a pre-tested study procedure (Johnson et al., 2013a), and the large sample size. Although self-reported alcohol consumption via computers has been shown to be reliable (Bonevski et al., 2010; Lotfipour et al., 2012), the main limitation of this study is the reliance on self-report, especially since many participants completed the AUDIT-C in close proximity to other people. Another limitation was the non-collection of alcohol consumption data from 40% of the adult hospital outpatients we invited.
Smoking among dental students at King Saud University: Consumption patterns and risk factors
2014, Saudi Dental JournalCitation Excerpt :One limitation of this study is related to the issue of validity for self-reported information, a common issue for surveys. Self-reported smoking status can agree with biomarker tests for tobacco use (Wong et al., 2012; Bonevski et al., 2010; Chen et al., 2002), but in some circumstances, such as with teenagers or pregnant women, self-reported smoking data may not be valid (Caraballo et al., 2004; Fakhfakh et al., 2011). Anonymous surveys or online surveys similar to the one used in this study generally have high reliability and validity (Ramo et al., 2011; Klein et al., 2007).
Risk factors for intracerebral hemorrhage differ according to hemorrhage location
2012, NeurologyCitation Excerpt :Furthermore, self-report does not account for the severity of the hypertension or hypercholesterolemia. Sensitivities for self-report of hypertension, hypercholesterolemia, tobacco use, and alcohol consumption in prior studies ranged from 51%–86%, and specificities ranged from 57%–95%.36–38 Cases and controls had risk factor assessment by the same methodology, however, and common atherosclerotic risk factors such as diabetes and cigarette smoking did not show significant differences between cases and controls, which suggests that recall bias was not a significant factor in our analysis.
Effect of alcohol consumption on breast cancer: probabilistic bias analysis for adjustment of exposure misclassification bias and confounders
2023, BMC Medical Research MethodologyCan breast cancer be stopped? Modifiable risk factors of breast cancer among women with a prior benign or premalignant lesion
2021, International Journal of Cancer