Article Text
PDF
Abstract
Objective To assess the performance of cardiovascular disease (CVD) risk equations in Indigenous Australians.
Methods We conducted an individual participant meta-analysis using longitudinal data of 3618 Indigenous Australians (55% women) aged 30–74 years without CVD from population-based cohorts of the Cardiovascular Risk in IndigenouS People(CRISP) consortium. Predicted risk was calculated using: 1991 and 2008 Framingham Heart Study (FHS), the Pooled Cohorts (PC), GloboRisk and the Central Australian Rural Practitioners Association (CARPA) modification of the FHS equation. Calibration, discrimination and diagnostic accuracy were evaluated. Risks were calculated with and without the use of clinical criteria to identify high-risk individuals.
Results When applied without clinical criteria, all equations, except the CARPA-adjusted FHS, underestimated CVD risk (range of percentage difference between observed and predicted CVD risks: −55% to −14%), with underestimation greater in women (−63% to −13%) than men (−47% to −18%) and in younger age groups. Discrimination ranged from 0.66 to 0.72. The CARPA-adjusted FHS equation showed good calibration but overestimated risk in younger people, those without diabetes and those not at high clinical risk. When clinical criteria were used with risk equations, the CARPA-adjusted FHS algorithm scored 64% of those who had CVD events as high risk; corresponding figures for the 1991-FHS were 58% and were 87% for the PC equation for non-Hispanic whites. However, specificity fell.
Conclusion The CARPA-adjusted FHS CVD risk equation and clinical criteria performed the best, achieving higher combined sensitivity and specificity than other equations. However, future research should investigate whether modifications to this algorithm combination might lead to improved risk prediction.
- epidemiology
- cardiac risk factors and prevention
- diabetes
- global health
- coronary artery disease
Statistics from Altmetric.com
Introduction
Cardiovascular disease (CVD) mortality rates have declined globally especially for regions with high overall socioeconomic advantage.1 In Australia, Aboriginal and Torres Strait Islander First Nations Indigenous peoples have experienced reductions in CVD mortality; however, there are still opportunities for improvement.2 Premature CVD contributes significantly to the large life expectancy gap between Indigenous and non-Indigenous Australians, and high rates of CVD coincide with diabetes and kidney disease.2
Primary prevention of CVD is underpinned by identification of modifiable CVD risk,3 4 and this is best achieved using risk prediction algorithms.4 In Australia, the National Vascular Disease Prevention Alliance, the Central Australian Rural Practitioners Association (CARPA) and the National Aboriginal Community Controlled Health Organisation/Royal Australian College of General Practitioners recommend a two-phase risk assessment. First, individuals are stratified as having a high CVD risk according to clinical criteria, which may identify up to 80% of all those classified as having high CVD risk.5 Second, among those not classified as high risk, the 1991 Framingham 5-year CVD risk equation is recommended (online supplementary eAppendix 3). However, the Framingham risk equation may underestimate CVD in Indigenous Australians.6 7 Consequently, CARPA recommended adding 5 percentage points to an individual’s risk in those not meeting the clinical criteria.8 However, implementation of these guidelines is inconsistent, and this algorithm’s clinical performance has not been evaluated with respect to CVD outcomes.
Supplemental material
Performance of CVD risk equations has mainly been assessed in populations of European and Asian heritage, with no head-to-head comparisons in Indigenous populations. Cardiovascular Disease Risk in IndigenouS People (CRISP) is a consortium of population-based longitudinal cohort studies of Indigenous Australians. Here we aimed to assess the predictive accuracy and clinical performance of six CVD risk equations in the CRISP populations, including: (1) the 1991 Framingham Heart Study (FHS) risk equation (1991-FHS)9; (2) the 1991 FHS risk equation after adding 5 percentage points to individual predicted risks according to CARPA guidelines8 (CARPA-adjusted FHS); (3) the 2008 FHS CVD risk equation (2008-FHS)10; (4) the American Heart Association (AHA)/American College of Cardiology (ACC) pooled cohorts (PC) equation for non-Hispanic whites (PC-NHW); (5) the AHA/ACC PC equation for African-Americans (PC-AA)3; and (6) the GloboRisk equation.11 12 We also assessed diagnostic accuracy when the equations were applied as they are currently recommended in clinical practice in Australia, reclassifying participants in accordance with the clinical risk criteria currently recommended (online supplementary eAppendix 3).
Methods
Participants
Five cohorts were selected based on similarity of age range, measurement of CVD risk factors, the timing of the baseline examinations and Indigenous community approval. The cohorts (online supplementary eAppendix 2) were the Perth Aboriginal Atherosclerosis Risk Study (PAARS; baseline examination 1998–1999), the Darwin Region Urban Indigenous Diabetes study 2003–2005), the Bathurst Island Renal Disease Study (BIRDS; 1992–2005), the Galiwin’ku Healthy Lifestyle Study (GHLS; 2001–2005) and the Well Person Health Check Study (WPHC; 1998–2007). Data were restricted to match the age used to derive these equations (30-74 years for all Framingham-based equations and 40–74 years for PC and GloboRisk). Participants were excluded if they have had a hospitalisation for a CVD event (as defined in online supplementary eAppendix 4) before their baseline examination or had missing data (online supplementary eAppendix 2). All participants provided informed consent, and ethics approvals for follow-up were obtained.
Risk model calculations
Mean (95% CI) observed and predicted risks were determined for the total cohort and by sex, baseline age group and diabetes status. Five-year CVD predicted risks were calculated for each participant according to the 1991-FHS9 and the CARPA-adjusted FHS8 equations, and 10-year CVD predicted risks were calculated for the 2008-FHS,10 the PC-NHW, the PC-AA3 and the laboratory-based GloboRisk score recalibrated for Australians which, unlike the other risk scores evaluated, accounts for the age-related attenuation of the relationships between risk factors and CVD outcomes.11 12 The 2008 Framingham and PC equations were recalibrated (online supplementary eAppendix 3). The 1991 Framingham equation does not permit recalibration. For GloboRisk, we used the Australian equation (online supplementary eAppendix 1). Predictor variables included age, total cholesterol, high-density lipoprotein (HDL) cholesterol, systolic blood pressure, self-reported current smoking and diabetes (self-reported and/or medical record reported diagnosis or a fasting plasma glucose ≥7.0 mmol/L) (online supplementary eAppendix 1).
Cardiovascular outcomes
The first observed fatal or non-fatal CVD event was analysed. Non-fatal CVD hospitalisations were obtained from hospital databases, and CVD deaths were obtained from the National Death Index. Participants not matched to these databases were considered not to have had a CVD event and to be alive. The International Classification of Diseases (ICD) version 10 (and equivalent ICD-9) codes were used to define CVD events according to the definitions reported for each equation (online supplementary eAppendix 4). The PC-NHW, PC-AA and GloboRisk equations predict atherosclerotic CVD (ASCVD) (myocardial infarction, coronary artery disease deaths and stroke). FHS-based equations also include heart failure and peripheral artery disease in defining CVD. Follow-up time was censored at: (1) 5 years (1991-FHS and the CARPA-adjusted FHS) and (2) 10 years (2008-FHS, the PC-NHW, PC-AA and GloboRisk).
Statistical analysis
Analyses were based on participants with complete data. Repeating the analyses using multiple imputation of baseline data showed similar findings (online supplementary eAppendix 3, table 11–12). Large sample tests for two proportions were used to compare observed and predicted CVD risks with 95% CI estimated from the binominal distribution. Calibration was assessed by plotting the observed proportions against predicted probabilities for each decile of predicted risk. Accuracy was assessed with the Brier score (average squared deviation between predicted and observed risk, with a lower score representing higher accuracy)13 and discrimination by calculating the area under the receiver operating characteristic curve statistic. Sensitivity, specificity and their 95% CI derived from the binominal distribution were calculated using the 15% risk threshold for the FHS equations and the 7.5% risk threshold for the PC equations, before and after reclassifying individuals meeting the high risk clinical criteria.
We undertook two sensitivity analyses. First, analyses were repeated among those not meeting high risk clinical criteria as this group would be recommended for CVD risk assessment using the Framingham equation under the Australian guidelines (online supplementary eAppendix 3). Second, to assess the impact of calendar time, analyses were stratified by year of baseline data collection (before and after the year 2000). All statistical analyses were undertaken in Stata V.15.0.
Results
Population characteristics
For the Framingham-based equations, there were 4681 participants aged 30–74 years at baseline, 358 had a prior CVD event and 732 had incomplete data and were thus excluded. The analysis was based on 1634 men and 1957 women. Compared with those included in the analysis, those with missing data had very similar age and sex distribution (online supplementary eTable 6, eAppendix 2). For the PC and GloboRisk equations, there were 2909 participants aged 40–74 years at baseline. Of these, 197 with a prior CVD event and 434 with incomplete data were excluded. Thus, these analyses were based on 992 men and 1286 women. Mean risk factor levels for total cholesterol and systolic blood pressure were similar across cohorts. GHLS showed lower mean HDL cholesterol (table 1). Diabetes prevalence ranged from 17% in BIRDS men to 39% in WPHC women. Smoking rates ranged from 40% in PAARS women to 80% in BIRDS men (detailed information on risk factors are in online supplementary eTable 8–10 and eFigures 1–3).
Description and characteristics of included studies* in the Cardiovascular Risk in IndigenouS People (CRISP) Consortium
CVD rates
Of the 3591 participants, 359 had a CVD event over 5 years, and 625 had an event over 10 years; a 10-year observed CVD event rate of 20.2 (95% CI 18.6 to 21.8) per 1000 person-years (18.2 (95% CI 16.3 to 20.4) per 1000 person-years for women and 22.5 (95% CI 20.2 t o25.2) per 1000 person-years for men). Of the 2278 participants aged 40–74 years, there were 339 ASCVD events over 10 years; a 10-year ASCVD event rate of 17.4 (95% CI 15.6 to 19.4) per 1000 person-years (15.6 (95% CI 13.4 to 18.1) for women and 19.8 (95% CI 17.0 to 23.1) for men).
Calibration
Observed and predicted CVD and ASCVD probabilities are outlined in table 2. When applied without clinical criteria, all equations, except for the CARPA-adjusted FHS, underestimated the observed risks. Underestimation was generally greater in women (except for the PC-NHW and PC-AA), younger age groups and among those without diabetes. For the prediction of CVD, the CARPA-adjusted FHS equation was well calibrated overall, although it significantly overestimated risk in those aged 30–39 years and those without diabetes. The 2008-FHS showed better calibration than the 1991-FHS in predicting CVD. For the prediction of ASCVD, the PC-NHW showed slightly better calibration than the PC-AA. GloboRisk showed the greatest underestimation. We generally observed underestimation of CVD in all strata (figure 1). Although the CARPA-adjusted FHS equation showed better calibration at higher predicted risk, it overestimated risk in women, those without diabetes or aged <45 years with lower predicted risk, and in men with higher predicted risk. In men with higher predicted risks, the 1991-FHS equation showed good agreement. For those with diabetes or aged ≥45 years, CARPA-adjusted FHS showed good agreement. When the equations were applied to the population who did not meet the high risk clinical criteria, the 1991-FHS, 2008-FHS and GloboRisk equations significantly underestimated risk and the CARPA-adjusted FHS equation significantly overestimated risk. The PC-NHW and PC-AA equations generally demonstrated better calibration in this lower risk cohort (table 3).
Observed and predicted probabilities using different CVD equations among the total population without CVD at baseline
Calibration plots of the observed versus the predicted CVD risk for six CVD risk equations in Indigenous men and women from the CRISP study. Points on the figure represent the mean observed (y-axis) and predicted (x-axis) risks according to deciles of predicted risk. Lines connect each of the data points. The CVD risk equations assessed for the population of non-pregnant participants aged 30–74 years include: Framingham Heart Study (FHS) 1991 and 2008 equations and Central Australian Rural Practitioners Association adjusted 1991-FHS equation (CARPA-adj FHS). The ASCVD risk equations assessed for the population of non-pregnant participants aged 40–74 years include: American Heart Association/American College of Cardiology (AHA/ACC) pooled cohorts non-Hispanic white (PC-NHW); AHA/ACC pooled cohorts African-American (PC-AA) and GloboRisk equations. ASCVD, atherosclerotic CVD; CRISP, Cardiovascular Risk in IndigenouS People; CVD, cardiovascular disease.
Observed and predicted probabilities using different CVD equations among the population who had low clinical risk according to NVDPA and CARPA guidelines
Discrimination and accuracy
When applied without the clinical criteria, the CARPA-adjusted FHS equation showed superior accuracy to the 1991-FHS in predicting 5-year risk, while the PC equations showed the greatest accuracy for prediction of 10-year risk. Overall, discrimination ranged from 0.66 for GloboRisk to 0.72 for all FHS-based equations. Accuracy and discrimination were better in women, in younger age groups and in those without diabetes (table 4). When the equations were applied as per current clinical practice in Australia, discrimination ranged from 0.61 for GloboRisk to approximately 0.69 for FHS equations (online supplementary eTable 16).
Statistical performance of the CVD risk equations in the CRISP cohorts
Diagnostic accuracy according to thresholds for clinical intervention
Among those who had a CVD event over 5 years, 31% on the 1991-FHS equation and 44% on the CARPA-adjusted FHS were classified as high risk at baseline when only using the risk equations (table 5). Using risk equations and clinical criteria to classify individuals into high-risk groups, sensitivity increased to 58% for 1991-FHS and 64% on the CARPA-adjusted FHS. Among those who had a CVD event over 10 years, 57% on the 2008-FHS, 80% on the PC-NHW and 79% on the PC-AA were classified as high risk on equations only. After using equations and clinical criteria to determine high risk, the proportions increased to 68% on the 2008-FHS, 87% on the PC-NHW and 86% on the PC-AA. The higher sensitivity was offset by low specificity, but the CARPA-adj FHS and 2008-FHS achieved both high sensitivity and specificity.
Diagnostic accuracy of CVD risk equations at clinical thresholds for medical intervention
Discussion
Our study assessed the performance of six frequently used CVD risk equations in Indigenous Australians. Overall, our findings indicate that when applied without the clinical criteria, none of the equations performed well on all assessment criteria, though CARPA-adjusted FHS was the most suitable overall. First, most 5-year and 10-year equations underestimated CVD risk. Although the CARPA-adjusted FHS equation minimised differences between the observed and predicted risks overall, it overestimated risk among lower risk groups. Second, all equations were only able to moderately discriminate between those with and without CVD. Third, CARPA-adjusted FHS and 2008-FHS were superior in terms of highest sensitivity and specificity according to high risk thresholds. When the clinical criteria were used with risk equations in accordance with Australian guidelines, sensitivity improved but specificity reduced. Under the CARPA-adjusted FHS algorithm (incorporating the adjusted FHS equation and clinical criteria), around one-third of Indigenous Australians who subsequently had a CVD event would not be recommended for primary CVD preventive medical treatment. Under the 1991-FHS equation, this figure increased to 40% and reduced to around 15% for the PC equations.
Several studies have shown that correction for population differences related to risk factor levels and incidence of CVD improves the calibration (or risk estimation) of CVD risk equations.7 14 In our study, despite recalibration, FHS and PC equations still substantially underestimated risk, possibly due to differences in the relationships between risk factors and CVD in this population. Indigenous Australians experience high risk factor levels and at younger ages. Our results showed that in those aged under 45 years, andbetween age and smoking. Interestingly, underestimation was greater with the GloboRisk equation, which accounts for the attenuation in the relationships between risk factors and CVD with age. It is possible that in this higher risk population, the need to account for this age-related attenuation is not as important. Risk equations that include interactions between age and diabetes may be required. Moreover, estimation of lifetime risk may also be important among younger Indigenous people who score a low risk as a result of their age despite CVD risk factors being present.
Risk algorithms should also be able to accurately discriminate between those who will and will not develop CVD. Our findings showed that when applied without clinical criteria, the equations demonstrated moderate discrimination (0.66–0.72), which was lower than that reported for the general Australian population (0.75–0.84).15 Discrimination may be improved through the addition of other risk factors to the equation. Chronic kidney disease (CKD) and/or albuminuria, which have also been associated with low birth weight16 and chronic inflammation,17 are highly prevalent among Indigenous Australians and are key drivers of premature mortality and the development of CVD.18 19 Thus, current CVD risk guidelines for Indigenous Australians attempt to capture the CVD risk associated with CKD by automatically classifying individuals with diabetes and microalbuminuria or those with at least moderate to severe CKD as high risk for CVD. Albuminuria is included in a coronary heart disease risk prediction score for Native Americans, who have similar risk profiles with regards to diabetes and CKD as Indigenous Australians,20 and findings from one cohort of CRISP has shown that a CVD risk score that includes albuminuria demonstrated better performance than a FHS risk equation.21 We found that when risk equations and clinical criteria were used as outlined in current clinical guidelines to define high-risk groups, a greater proportion of individuals who had CVD events were captured by the CVD risk algorithm, but this improvement was somewhat off-set by reduced specificity.
In many Indigenous populations, including Indigenous Australians, socioeconomic deprivation is a key determinant of health and well-being, and risk equations that do not include a measure of social deprivation underestimate risk in those from lower socioeconomic groups.22 Despite this, few CVD risk equations include a measure of social deprivation.3 23 Future work should assess the contribution of measures of socioeconomic status to CVD risk prediction in Indigenous Australians.
The following limitations should be considered. First, sensitivity and specificity analyses were based on the risk level recommended for the initiation of medical therapy. However, we recognise that under Australian guidelines medical therapy is also recommended to those at moderate risk if after six months, lifestyle modifications fail to reduce predicted risk.24 Therefore, we may be overestimating the proportions of people missed for medical intervention. Second, Indigenous Australians living in large cities in Australia were under-represented.25 Third, participants were volunteers, and thus our results may be subject to healthy volunteer biases. Fourth, it was not feasible to adjudicate CVD from medical records. Inadequate identification of CVD events prior to baseline and misclassification of CVD during follow-up may have resulted in a higher risk cohort. Fifth, stratification may have resulted in greater imprecision due to the smaller number of CVD events in some groups. Sixth, we did not have reliable data on medication use, and thus we applied the beta-coefficient for untreated systolic blood pressure in calculating predicted risks using the 2008 FHS and PC equations. However, beta-coefficients for treated and non-treated BP were similar in these equations, and using the treated BP coefficients led to similar findings (data not shown). Seventh, the baseline data were collected at different time points and may therefore represent different prevalence estimates of CVD risk factors. However, findings were similar when stratified by calendar year of baseline data collection (online supplementary eTables 13–14). Although rates of smoking and diabetes prevalence in the CRISP study populations were similar to those reported in the 2004–2005 National Aboriginal and Torres Strait Islander Health Survey,26 risk factor levels in CRISP are higher than those reported for the 2012–2013 National Aboriginal and Torres Strait Islander Health Survey,27 which may have led to a greater degree of underestimation than would be observed in a lower risk contemporary population. Finally, given that risk factor prevalence varies according to region,27 it may be important to account for this when updating risk algorithms for Indigenous Australians.
Underestimation of CVD risk in Indigenous Australians is compounded by low rates of screening and medical management of the at-risk population.28 However, rapid improvement in population coverage for risk assessment is possible by including automated risk assessment within the electronic medical record and reporting screening rates to frontline primary health teams to proactively close evidence-practice gaps in care.29 Implementation of electronic decision support tools and primary healthcare performance reporting should be explored to improve the use of risk equations for Indigenous Austracalians.
Conclusion
Overall, recalibrated CVD risk equations developed from the FHS and the AHA/ACC pooled cohorts when applied without clinical criteria underestimate CVD risk in Indigenous Australians with high prevalence of CVD risk factors. Out of the equations assessed, the CARPA-adjusted FHS CVD equation in conjunction with the clinical criteria was most suitable overall compared with the other equations. However, further research should investigate whether CVD risk prediction can be improved for Indigenous Australians. Strategies might involve developing new risk equations and/or clinical algorithms that better capture the risks associated with chronic kidney disease and the early age of onset of diabetes.
Patient and public involvement
Indigenous community approvals were sought to include deidentified data from each of the cohorts included in the CRISP collaboration and to ensure that the research questions met their priorities. ST, LD and AB are Aboriginal researchers and CRISP study investigators on this manuscript. They were involved in the study design, interpretation and manuscript drafts and will lead dissemination of study findings. The CRISP study investigators will work with Indigenous community representatives from each of the cohorts to write a plain language summary and provide culturally acceptable feedback to their communities.
Key messages
What is already known on this subject?
In Australia, premature cardiovascular disease (CVD) significantly contributes to the estimated 10-year life expectancy gap between Indigenous and non-Indigenous Australians. Head-to-head assessment of commonly used risk equations in conjunction with current clinical guidelines has not been undertaken in Indigenous Australians.
What might this study add?
Our findings support the use of the Central Australian Rural Practitioners Association (CARPA)-adjusted CVD risk algorithm in Indigenous Australians. However, its identified limitations show that future research should investigate whether modifications to this algorithm might improve CVD risk prediction.
How might this impact on clinical practice?
The CARPA-adjusted FHS CVD risk algorithm that incorporates clinical criteria and risk equation is recommended for Indigenous Australians. These findings may have implications for other populations who have a similar high risk profile.
Acknowledgments
The authors gratefully acknowledge the support of study participants, study staff and partner organisations from each of the cohorts included in the Cardiovascular Risk in IndigenouS People (CRISP) Consortium.
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Correction notice Since this article was first published online, Charles Darwin University has been added to the first affiliation.
Contributors ELMB had full access to all the data in the study and takes responsibility for its integrity and the data analysis. ELMB led all aspects of the conduct of the study and designed the study, drafted the manuscript, collected, analysed and interpreted the data; FB analysed and interpreted the data and revised the manuscript; AR collected and analysed the data and reviewed the manuscript; JC, ST, RMcD, WEH, ZW, PJB, PLT, LD, JB and KO'D collected the data, interpreted the data and revised the manuscript; CC, PB, SG, AB and AC interpreted the data and revised the manuscript; and JS and LM-B conceived and designed the study, interpreted the data and revised the manuscript. All authors approved the final version. The corresponding author has the right to grant on behalf of all authors and does grant on behalf of all authors, an exclusive licence on a worldwide basis to the BMJ Publishing Group Ltd and its Licensees to permit this article (if accepted) to be published in HEART editions and any other BMJPGL products to exploit all subsidiary rights.
Funding The CRISP consortium has received funding from the National Heart Foundation of Australia (Vanguard grant #100595) and a National Health and Medical Research Council (NHMRC) programme grant (#631947). ELMB was supported by a National Heart Foundation postdoctoral fellowship (#101291). LM-B was supported by NHMRC Fellowship (#605837) and NHMRC Practitioner Fellowship (#1078477). JC was supported by an NHMRC Research Fellowship (#1058244). JS was supported by a NHMRC Senior Research Fellowship (#1079438).
Disclaimer The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.The views expressed in this publication are those of the authors and do not reflect the views of the NHMRC.
Competing interests None declared.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement Data are available on reasonable request. Requests for data need to be sought from CRISP study investigators, communities who contributed data and relevant ethics committees. All requests are to be sent to Dr Elizabeth Barr (corresponding author) at Elizabeth.Barr@menzies.edu.au.