Abstract
Background Despite its rarity, the prognosis of pancreatic cancer is very poor and it is a major cause of cancer mortality; being ranked fourth in the world, it has one of the worst survival rates of any cancer.
Aim To evaluate the performance of QCancer® (Pancreas) for predicting the absolute risk of pancreatic cancer in an independent UK cohort of patients, from general practice records.
Design and setting Prospective cohort study to evaluate the performance QCancer (Pancreas) prediction models in 364 practices from the UK, contributing to The Health Improvement Network (THIN) database.
Method Records were extracted from the THIN database for 2.15 million patients registered with a general practice surgery between 1 January 2000 and 30 June 2008, aged 30–84 years (3.74 million person-years), with 618 pancreatic cancer cases. Pancreatic cancer was defined as incident diagnosis of pancreatic cancer during the 2 years after study entry.
Results The results from this independent and external validation of QCancer (Pancreas) demonstrated good performance data on a large cohort of general practice patients. QCancer (Pancreas) had very good discrimination properties, with areas under the receiver operating characteristic curve of 0.89 and 0.92 for females and males respectively. QCancer (Pancreas) explained 60% and 67% of the variation in females and males respectively. QCancer (Pancreas) over-predicted risk in both females and males, notably in older patients.
Conclusion QCancer (Pancreas) is potentially useful for identifying undetected cases of pancreatic cancer in primary care in the UK.
INTRODUCTION
The incidence of pancreatic cancer is moderately low: it is the 13th most common cancer worldwide and the 9th most common in high-income countries.1 Despite its rarity, the prognosis of pancreatic cancer is very poor and it is a major cause of cancer mortality; being ranked fourth in the world,2 it has one of the worse survival rates of any cancer. In the UK in 2008, 8085 people were diagnosed with pancreatic cancer and 7781 died from this cancer.3 The 1-year survival rate is less than 20%, while the 5-year survival rate is only around 3%.3 The difficulty of early detection contributes to the poor prognosis in patients with pancreatic cancer, with most patients going undiagnosed until emergency admission. Pancreatic cancer has very few symptoms until the disease is at a relatively advanced stage, when curative treatment is limited. There are currently very few reliable tools to identify individuals at an increased risk of having or developing pancreatic cancer.4,5
QCancer® (Pancreas) is a pair of multivariable prediction models (one for males and one for females) that have been developed recently to predict the risk of having undiagnosed pancreatic cancer.6 QCancer (Pancreas) was developed and internally validated on a large cohort of 3.6 million patients from the QResearch (www.qresearch.org) database.6 This is a large database comprising over 12 million anonymised health records from 602 general practices throughout the UK, using the EMIS computer system. The models were derived on 2.4 million patients aged between 30 and 84 years, contributing 1415 incident cases of pancreatic cancer from 4.1 million person-years of observation between 1 January 2000 and 30 September 2010. The final prediction models, based on a Cox proportional hazards model, included eight risk factors for females and nine risk factors for males (Table 1). Open source code to calculate the QCancer (Pancreas) score is available from www.qcancer.org/, released under the GNU Lesser General Public License, version 3. The performance of QCancer (Pancreas) was assessed on separate sample of 1.2 million patients (781 incident cases of pancreatic cancer) from the same QResearch database (different general practices), with good discriminative ability and calibration.6
Risk factors in QCancer® (Pancreas)
QCancer (Pancreas) is part of a suite of prediction models that form the QCancer scores (www.qcancer.org) that have been developed to predict the risk of having undiagnosed lung,7 ovarian,8 colorectal,9 gastro-oesophageal,10 renal,11 or pancreatic cancer.6 The authors are currently using identical methods to independently evaluate the performance of these six predictions models. To date, they have published validation studies for QCancer (Colorectal),12 QCancer (Ovarian),13 QCancer (Gastro-Oesophageal),14 and QCancer (Renal),15 to predict the risk of undiagnosed colorectal, ovarian, gastro-oesophageal and renal cancer, respectively. The description of the methods used within this study are thus substantially the same as the first study published, evaluating QCancer (Colorectal).12
This article describes the results from an independent evaluation of QCancer (Pancreas) on a large dataset of general practice records in the UK that were not used to derive the prediction model.
How this fits in
The prognosis of pancreatic cancer is poor and is a major cause of mortality. Contributing to the poor prognosis is that pancreatic cancer has very few symptoms until the disease is relatively advanced. A new diagnostic prediction model (QCancer® [Pancreas]) has recently been developed to predict the risk of undiagnosed pancreatic cancer in a primary care population. The prediction model has good predictive ability and could be integrated into clinical computer systems to identify individuals who are at increased risk of having undiagnosed pancreatic cancer.
METHOD
Cohort selection
Study participants were patients registered between 1 January 2000 and 30 June 2008 and recorded on The Health Improvement Network (THIN) database (www.thin-uk.com). The same exclusion criteria as in the original development paper were adopted.6 Patients were excluded if they had a prior diagnosis of pancreatic cancer, were registered for <12 months with the general practice, had invalid dates, were aged ≤30 years, or ≥85 years. Entry to the cohort was defined as for the original development study,6 as the latest of (1) the study start date; (2) the date the patient registered with the practice; and (3) for those patients with red flag symptoms (abdominal pain, appetite loss, dysphagia, weight loss, abdominal distension or constipation), the date of the first recorded onset of any red flag symptom within the study period.
Outcome measure
The outcome was diagnosis of pancreatic cancer as described in the original study that developed QCancer (Pancreas),6 defined as incident diagnosis of pancreatic cancer during the 2 years after study entry. The only difference between this external validation and the original development of QCancer (Pancreas) is that there is no linkage to Office of National Statistics (ONS) death records, and thus the outcome was based solely on what was recorded on the patients’ records (THIN). Patients not experiencing the study outcome were censored at the earliest date: either date of death, date of leaving the practice study, or 2 years of follow-up.
Statistical analysis
The 2-year predicted risk of pancreatic cancer was calculated for each patient in the THIN cohort using the QCancer (Pancreas) prediction model (www.qcancer.org/). Multiple imputation, using all predictors plus the outcome variable and censoring status, was used to replace missing values for smoking status. This involves creating multiple copies of the data and imputing the missing values with sensible values randomly selected from their predicted distribution. Ten imputed datasets were generated and results from analyses on each of the imputed datasets were combined using Rubin’s rules to produce estimates and confidence intervals that incorporate the uncertainty of imputed values.16 Smoking status was derived from combining two risk factors: (1) whether the patient was a non-smoker, ex-smoker or current smoker; and (2) the amount of cigarettes smoked, defined as light (<10 cigarettes/day), moderate (10–19 cigarettes/day), or heavy (≥20 cigarettes/day).
Predictive performance of the QCancer (Pancreas) prediction model on the THIN cohort was assessed by examining measures of calibration and discrimination. Calibration refers to how closely the predicted 2-year pancreatic cancer risk agrees with the observed proportions of patients with a diagnosis of pancreatic cancer within 2 years. This was assessed for each tenth of predicted risk, ensuring 10 equally sized groups, and each 5-year age band, by plotting observed proportions versus predicted risk.
Discrimination is the ability of the prediction model to differentiate between patients who experience an event during the study period and those who do not. This measure is quantified by calculating the c statistic: a value of 0.5 represents chance and 1 represents perfect discrimination.17 The D statistic18 and R2 statistic19 were also calculated; these are measures of discrimination and explained variation, respectively, and are tailored towards censored survival data. The D statistic is a measure of prognostic separation of survival curves and is closely related to the standard deviation of the prognostic index (the linear component from the Cox model). R2 (explained variation) is the proportion of total variation in the outcome that is explained by the prediction model, ranging from 0 to 100%.
All statistical analyses were carried out in R (version 2.14.1, http://www.R-project.org) and the ICE (multiple imputation) procedure in Stata (version 11.2).
RESULTS
Between 1 January 2000 and 30 June 2008, 2 150 322 eligible patients from 364 general practices in the UK were registered in the THIN database. The 2 150 322 eligible patients contributed 3 744 567 person-years of observation (median follow-up was 2 years), among whom there were 618 cases of pancreatic cancer (287 females; 331 males). Table 2 details the characteristics of eligible patients. Compared with the original development cohort (QResearch),6 members of the THIN cohort were marginally younger, proportionally more patients reported abdominal pain (11.9% versus 9.8%), and there were fewer non-smokers (40.1% versus 50.8%).
Characteristics of participants aged 30 to 84 years in the QResearch development and THIN validation cohorts
Complete data on smoking status were available for 80.6% of females (n = 873 026) and 77.2% of males (n = 823 873). There were noticeably more current smokers in the THIN cohort, for whom the number of cigarettes was not recorded (13.2% in THIN compared to around 3% in the original QResearch database).
Table 3 reports the age–sex incidence rates of each symptom included in the QCancer (Pancreas) prediction models. All the symptoms apart from abdominal pain in females tended to become more common with age. During the follow-up, the crude rate of pancreatic cancer was 17 per 100 000 person-years of observation (compared with 30 per 100 000 in the original QResearch cohort), with 18 per 100 000 person-years for females and 15 per 100 000 person-years for males. As in the original development cohort, incidence rates of pancreatic cancer increased sharply with age.
Incidence rates of appetite loss, weight loss, abdominal pain, abdominal distension, and dysphagia per 100 000 person-years by sex and age in the THIN cohort
Performance data for QCancer (Pancreas) from the original development cohort and the THIN cohort (multiple imputation and complete-case) are presented in Table 4. The values for the R2 statistics (percentage of explained variation) and the D statistic were marginally higher in the THIN cohort, 60.0% and 2.51 (females) and 66.6% and 2.89 (males), compared to those reported in the original development paper: 58.7% and 2.44 (females) and 62.0% and 2.61 (males). These high values of the D statistic and also of the c statistic, 0.89 (females) and 0.92 (males), indicate good discrimination properties of QCancer (Pancreas). The value of using individual symptoms to identify patients with pancreatic cancer is compared to QCancer (Pancreas) in Table 5. Using a risk cut-off of 0.2% that identified the 10% of females and males at the highest risk, QCancer (Pancreas) clearly showed an improvement, with 64% and 74% of all new pancreatic cancers identified for females and males respectively. Using the presence of abdominal pain, 54% and 60% of new pancreatic cancers were identified in females and males respectively. The sensitivity of the other individual symptoms ranged between 2.5% (abdominal distention) and 9.1% (weight loss) in females and between 2.7% (dysphagia) and 16.9% (weight loss) in males.
Performance data
Comparison of strategies to identify patients having a diagnosis of pancreatic cancer in the next 2 years
Calibration plots of QCancer (Pancreas) for females and males by tenth of risk are presented in Figure 1. QCancer (Pancreas) increasingly over-predicts risk, with increases across the tenths of risk. Similarly, Figure 2 displays the calibration plots of QCancer (Pancreas) for females and males by 5-year age bands. Again, the QCancer (Pancreas) systematically over-predicts risk across all age groups in both females and males.
Observed versus predicted pancreatic cancer risks (triangles denote predicted risk, rectangles denote observed risk).
Observed versus predicted pancreatic cancer risks by sex and age (triangles denote predicted risk, rectangles denote observed risk).
DISCUSSION
Summary
QCancer (Pancreas) is a new prediction model to identify individuals with undetected pancreatic cancer in a primary care setting. The prediction model was developed and internally validated on a large primary care electronic database (QResearch) of 3.6 million patients, contributing 2196 new cases of pancreatic cancer between 1 January 2000 and 30 September 2010.
This independent evaluation of the performance of QCancer (Pancreas) was carried out on the large separate database (THIN), based on general practices recording clinical data using the INPS Vision system that is used in 20% of UK general practices. The database comprised 2.15 million patients between 1 January 2000 and 30 June 2008, contributing 3.7 million person-years of observation and 618 cases of pancreatic cancer. The performance data presented in this paper on the THIN cohort provide good evidence to support the external validity of QCancer (Pancreas) in identifying patients with suspected pancreatic cancer, with good discrimination performance that is marginally better than that observed in the internal validation data. QCancer (Pancreas) also clearly outperformed the use of individual symptoms for identifying new cases of pancreatic cancer.
Strengths and limitations
The calibration was disappointingly moderate, with QCancer (Pancreas) systematically over-predicting the risk of undetected cancer. There was a suggestion of this effect in the last three or four tenths of risk in the original study developing the model.6 The development cohort comprised 2.4 million patients and included only 1415 cases of pancreatic cancer, which arguably can be described as a rare event (despite there being more than the recommended 10 events per variable,20 and could contribute to lack of calibration observed in the external validation. The case-mix of patients who had undetected pancreatic cancer may also be different between the original development data set and the THIN external validation cohort, which may also contribute to the observed miscalibration. The slight difference in outcome definition also possibly contributes to the miscalibration, with the original study including linkage to death records, while this external validation did not include the linkage. The slight difference in outcome definition meant a difference in the proportion diagnosed (and died) of pancreatic cancer of 0.06% in the original development cohort compared to 0.03% in this external validation.
Comparison with existing literature
QCancer (Pancreas) are the first multivariable prediction models to predict the risk of undiagnosed pancreatic cancer for use in UK primary care. Clearly the usefulness of the prediction models is predicated on having accurate information recorded in primary care electronic healthcare records. However, studies have shown good agreement between diagnoses recorded in general practice databases and other data sources.21–24
Implications for clinical practice
To date, the development, internal validation, and this external validation of QCancer (Pancreas) has used 5.8 million patients, contributing 10 million person-years of observation and 2814 cases of pancreatic cancer during the observation periods to develop and evaluate QCancer (Pancreas), in order to predict the risk of pancreatic cancer in adults aged 30–84 years. This study has provided an independent and external evaluation of the QCancer (Pancreas) prediction model on a large cohort of patients in the UK. It has assessed the performance of QCancer (Pancreas) against performance metrics presented in the internal validation of QCancer (Pancreas) and has provided evidence to support the external validity of QCancer (Pancreas).
The performance of QCancer (Pancreas) was similar to that in the internal validation of QCancer (Pancreas), with comparable results indicating excellent discrimination of the prediction model. However, QCancer (Pancreas) over-predicted risk in both females and males, which may be attributed to a slight difference in outcome definition.
Acknowledgments
We thank Alyson Huntley for her assistance in review of data extraction.
Notes
Funding
None.
Ethical approval
Ethical approval was received from Trent multicentre research ethics committee.
Provenance
Freely submitted; externally peer reviewed.
Competing interests
The authors have declared no competing interests.
Discuss this article
Contribute and read comments about this article on the Discussion Forum: http://www.rcgp.org.uk/bjgp-discuss
- Received January 15, 2013.
- Revision received February 5, 2013.
- Accepted April 3, 2013.
- © British Journal of General Practice 2013
REFERENCES
In this issue
Jump to section
More in this TOC Section
Related Articles
Cited By...
