Abstract
Background Primary care is accessible and ideally placed for case finding of patients with lifestyle and mental health risk factors and subsequent intervention. The short self-administered Case-finding and Help Assessment Tool (CHAT) was developed for lifestyle and mental health assessment of adult patients in primary health care. This tool checks for tobacco use, alcohol and other drug misuse, problem gambling, depression, anxiety and stress, abuse, anger problems, inactivity, and eating disorders. It is well accepted by patients, GPs and nurses.
Aim To assess criterion-based validity of CHAT against a composite gold standard.
Design of study Conducted according to the Standards for Reporting of Diagnostic Accuracy statement for diagnostic tests.
Setting Primary care practices in Auckland, New Zealand.
Method One thousand consecutive adult patients completed CHAT and a composite gold standard. Sensitivities, specificities, positive and negative predictive values, and likelihood ratios were calculated.
Results Response rates for each item ranged from 79.6 to 99.8%. CHAT was sensitive and specific for almost all issues screened, except exercise and eating disorders. Sensitivity ranged from 96% (95% confidence interval [CI] = 87 to 99%) for major depression to 26% (95% CI = 22 to 30%) for exercise. Specificity ranged from 97% (95% CI = 96 to 98%) for problem gambling and problem drug use to 40% (95% CI = 36 to 45%) for exercise. All had high likelihood ratios (3–30), except exercise and eating disorders.
Conclusion CHAT is a valid and acceptable case-finding tool for most common lifestyle and mental health conditions.
INTRODUCTION
Increasing emphasis on preventive practice in primary healthcare necessitates identifying patients with lifestyle and mental health risk factors. Many at-risk behaviours and conditions may not be identified in routine practice at present. For example, the recent Mental Health in General Practice Investigation study reported that a third of primary care patients had experienced a diagnosable mental health disorder according to the Diagnostic and Statistical Manual of Mental Disorders, Fourth edition (DSM-IV).1 Despite the prevalence of mental health disorders presenting in primary care settings, the World Health Organization reports that many of these disorders go undiagnosed and they estimate that less than one-third of those who need treatment receive it.2
General practice is highly accessible to patients requiring help with problem behaviours, and patients expect to receive preventive lifestyle advice from their GP.3 Research shows that 80% of the population consult with their GP at least once a year.4 However, opportunistic screening is likely to have a limited effect5 and, given consultation time restraints, compliance with routine screening regimes can be low for both patients and practitioners.6,7
Some patients are embarrassed or object to being asked sensitive questions about their lives. For example, a number of studies examining women's acceptability of domestic violence screening show huge variability in the percentage of women who object – ranging from 15 to 57%.8 Results of such studies indicate there is a need for development of tools to help primary care better address this sector of practice for the population. Furthermore, any tool would have to be acceptable, reliable, and valid before widespread use.
The short self-administered Case-finding and Help Assessment Tool (CHAT) was developed for lifestyle and mental health assessment of adult patients (aged ≥16 years) in primary health care. The CHAT assesses for tobacco use, alcohol and other drug misuse, problem gambling, depression, anxiety and stress, abuse, anger problems, inactivity, and eating disorders. The tool was designed by a team of GPs, university researchers, a psychologist, and a community-based brief-intervention educator of primary healthcare providers.
The acceptability and feasibility of the CHAT has been assessed previously in 2543 patients of GPs and practice nurses in 41 rural and urban New Zealand general practices.9 Patients came from diverse ethnic, geographical, and socioeconomic backgrounds. The sample prevalence of positive responses ranged from 2.8% (gambling) to 42.7% at risk of depression. The number of patients requesting immediate assistance with these responses (from 0.5% for gambling through to 13.5% for depression and anxiety) did not overwhelm clinicians. All patients and practitioners in the evaluation study9 completed feedback forms, which recorded objections to any of the screened topics and positive and negative responses to the tool. The tool was well accepted by patients, with few objections to specific questions (0.1–0.8%) and was not considered overly burdensome by practitioners.
Given that the tool is largely based on previously reported measures and has undergone considerable practitioner and patient assessment, it can be assumed that it has good content validity.
Subsequent to this study, the CHAT has been used in a number of clinical settings, including the general practice of one of the co-authors, with no complaints reported.
While many of the individual items in the CHAT have been validated against more comprehensive tools, the CHAT had not been validated as a whole. The aim of this study was to conduct a criterion-based validation of the tool against a group of gold standard instruments.
How this fits in
Detecting lifestyle and mental health risk factors in primary healthcare patients can have valuable health outcomes, but frequently does not occur in the context of busy consultation schedules. The short self-administered Case-finding and Help Assessment Tool (CHAT) identifies tobacco use, alcohol and other drug misuse, problem gambling, depression, anxiety and stress, abuse, anger problems, inactivity, and eating disorders. It is well accepted by patients, GPs, and nurses. On validation against a composite gold standard, the CHAT has proved to be a valid and acceptable case-finding tool for most common lifestyle and mental health disorders.
METHOD
The tool was validated in primary healthcare practices in a primary health organisation in South Auckland (a socioeconomically deprived region) and a primary health organisation in the North Shore (a socioeconomically advantaged region) of Auckland.
The gold standard tools were selected using a pragmatic approach (Table 1). While DSM-IV diagnostic interviews might be ideal, conducting these in combination would have been too time-consuming to be practical in the primary healthcare setting; and requiring each participant to complete one diagnostic instrument on a random basis would have required a prohibitively large sample size.
Table 1 Gold standards used for each component of the CHAT.
All consecutive primary healthcare patients aged 16 years and over attending the practices were invited to complete the CHAT and a composite gold standard. Exclusion criteria were inability to understand English, or mental impairment that precluded meaningful participation. Recruitment ceased when 1000 patients had been recruited.
The CHAT and composite gold standard forms were self-administered by patients in the waiting room. There was a research assistant present to assist with consent and collection who was advised not to look at the screening tool answers when patients were completing the gold standard form. The study was conducted according to the Standards for Reporting of Diagnostic Accuracy (STARD) statement for diagnostic tests.10 Where the tool detected a risk factor that the patient wanted addressed, the GP could either deal with the problem at the time of the consultation or schedule a later appointment.
Data analysis was conducted using Microsoft Excel. Scores on the gold standard forms were dichotomised as ‘case’ or ‘not-a-case’. Sensitivities, specificities, positive and negative predictive values (PPV and NPV), and likelihood ratios were calculated using the online statistical calculator from the Centre for Evidence-Based Medicine (www.cebm.utoronto.ca/practise/ca/statscal/).
RESULTS
There was a 2% decline rate from consecutive eligible patients invited to participate. Sets of completed CHAT and gold standard forms were available from 995 patients, although not all CHAT questions and individual gold standard questionnaires were completed by all participants. Response rates ranged from 99.8% for smoking to 79.6% for anger questions (Table 2). Where response was incomplete, generally the CHAT was completed but not the relevant section of the composite gold standard.
Table 2 CHAT positive and negative likelihood ratios, condition prevalence, and response rate.
Sixty per cent of the participants were New Zealand European, 16% Maori, 4% Pacific people, and 20% Asian or ‘other’ ethnicity. This is representative of the New Zealand population. Seventy-one per cent were female, a proportion commonly found in general practice adult-attending patient populations. The age distribution was slightly skewed to the older age groups: 21% were aged 16–29 years, 37% were in the 30–49 year age bracket, and 42% were aged ≥50 years.
Sensitivities, specificities, PPV, and NPV are recorded in Table 3. PPVs ranged from 68% for nicotine dependency and 44% for problematic drinking, which were high prevalence conditions (14.7% and 12% respectively) to 1% for physical violence (a low prevalence condition of 0.6%). NPVs were all between 97 and 100%, except for the exercise question, which produced flawed results in all probability due to systematic error.
Table 3 CHAT sensitivity, specificity, positive and negative predictive values.
The positive and negative likelihood ratios (LR+ and LR−), condition prevalence, and response rates are reported in Table 2. Likelihood ratio incorporates both sensitivity and specificity and is a direct estimate of how much the test result changes the odds of having the condition. The questions for smoking, problematic drug use, gambling, and abuse all had a LR+ >10, which indicates that it is a very good test for ‘ruling in’ the condition. Alcohol dependency and major depression had LR− <0.1 which indicates that it is a very good test for ‘ruling out’ these conditions.
While the ‘eating disorder’ questions in the CHAT had good test properties to exclude an eating disorder (NPV = 98%; LR−= 0.14) there was low LR+ of 2.75, specificity (67%), with 14% ‘prevalence’. Given that less than 5% of primary healthcare patients are likely to meet DSM-IV criteria for eating disorders,1 it seems apparent that these questions were measuring something significantly broader than eating disorder itself.
DISCUSSION
Summary of main findings
The CHAT tool is both valid and acceptable for lifestyle and mental health disorder screening in primary care. All items showed good sensitivity, specificity, and likelihood ratios when compared with gold standard instruments, except for exercise and eating disorders. The validation of this tool used a pragmatic approach, given that a battery of gold standard tests needed to be administered in a waiting-room environment. This meant that brief, but well validated, tools such as the Alcohol Use Disorders Identification Test, Patient Health Questionnaire depression scale (PHQ-9), and the Hospital Anxiety and Depression Scale were used rather than longer instruments, such as a full Composite International Diagnostic Interview, which would have been too time-consuming and impractical in this setting.
The low response rates (79.6–84.2%) for the abuse and anger questions reflect reluctance of responders to complete the gold standard (Conflict Tactics Scale; CTS-1) for these items. This tool takes a long time to complete (and is therefore not appropriate for use in a waiting-room setting), and also asks particularly sensitive questions. The other lengthy gold standard was the Aerobics Center Longitudinal Study for exercise which was completed by only 87.8% of responders, whereas response rates for the other conditions were 91.4–99.8%.
Sensitivity and specificity of the single exercise question were extremely low (26% and 40% respectively), with the PPV only 27%, compared with 81% in a previous primary care study which validated it as a screening question for being sedentary.11 This was probably due to the way the question was presented in the CHAT, causing confusion to the responders. With all the other questions, a ‘yes’ response indicated a possible condition, whereas with the exercise response ‘no’ indicated probable sedentary behaviour. With the exception of the exercise question, ‘no’ responses were down the left-hand column of the tool. This was to aid the clinician by running the eye down the left-hand column to check for absence of conditions needing further enquiry. Examining the gold standard responses of those who ticked ‘no’ for the exercise question, it is apparent that many actually were very physically active, and it seems probable that the format caused them to invert their replies.
The eating disorder prevalence in this study was 14%, which is more than seven times higher than that reported in other studies.1 This study has demonstrated that the gold standard is not precise enough as it was probably identifying participants who had concerns about being overweight and eating patterns, rather than a formal eating disorder. As eating disorders are relatively rare in general practice, it is planned to remove this question from future screening tools.
Strengths and limitations of the study
The limitations of this study firstly are that some of the conditions had very small numbers of responders with the condition. Secondly, pragmatic gold standard instruments were used because administering diagnostic interviews for all conditions would have been excessively time-consuming to conduct in the primary healthcare setting. Some of the gold standards used are diagnostic tools and some are primarily screening instruments that have been validated against a gold standard, such as a DSM-IV interview. For example, while the South Oaks Gambling Screen might be considered a screening tool, there is no epidemiological gold standard in the area of disordered gambling prevalence.12 The South Oaks Gambling Screen was used as the gold standard as it has been the primary method used to identify problem and pathological gambling since the late 1980s.13
A strength of this study is that there was a very low decline rate and a consecutive group of patients. This means these results are generalisable to other general practices for consecutive patients. For the high prevalence conditions adequate numbers were found which produced narrow confidence intervals. A further strength is that patients completed the gold standards for all the conditions.
Implications for future research and clinical practice
The CHAT provides an important tool for routine use in primary healthcare settings in lifestyle and mental health domains where strong argument can be made for case finding and subsequent intervention.
It is envisaged that CHAT will be used in different ways in different settings. Some practices are already using it with all new patients, and asking adult patients to complete it if it has been more than 2 years since their last visit. Moves are under way to have the CHAT entered electronically and self-administered on a waiting-room touch screen where this is available. Because patients complete the CHAT before their consultation, they indicate whether or not it has brought up any issues that they wish to address in their consultation. This means that it is unlikely to inhibit or hinder a patient discussing their own agenda rather than their doctor's prevention agenda.
As well as being a self-administered waiting-room tool, the CHAT can also be administered by the GP and by other health professionals, such as the practice nurse. Once available electronically, the CHAT will be integrated into the patient's electronic record (patient management system). This will also allow for ‘second tier’ tools (for example the Alcohol Use Disorders Identification Test) to be available for administering from the practitioner's computer (including automatic scoring) should a patient have a positive response on the CHAT.
Because it is quick to use and well accepted, the CHAT can be used for follow-up after intervention for identified problems. The next step in the development of this tool is to conduct a trial to test against clinical outcomes. This will establish whether systematic use of the CHAT in primary healthcare setting leads to better health outcomes for patients.
Acknowledgments
The study involved collaboration between primary healthcare researchers with specific lifestyle or mental health interests and expertise in the Department of General Practice and Primary Health Care, the University of Auckland in the development of the tool. We thank all those who have made contributions to this work over the several years of its development
Notes
Funding body
The Oakley Mental Health Foundation (project grant 3606867) and the Charitable Trust of the Auckland Faculty of the Royal New Zealand College of General Practitioners (project grant 3601614)
Ethical approval
Approval was obtained from the Auckland Ethics Committee (AKY/04/04/079)
Competing interests
The authors have stated that there are none
- Received May 10, 2007.
- Revision received July 30, 2007.
- Accepted September 3, 2007.
- © British Journal of General Practice, 2008.
