Measuring continuity of care: psychometric properties of the Nijmegen Continuity Questionnaire

Participants and design

In The Netherlands, every patient is registered with a GP. The GP functions as a gatekeeper for secondary care, which is provided in a hospital/outpatient department. In a previous pilot study, the NCQ was distributed among patients with a chronic disease recruited from general practice.11 As the NCQ had been changed during the pilot testing, and it was wanted to test it in both primary and secondary care, another sample of participants was used.

In January 2010, 19 GP trainees working in practices in the eastern part of The Netherlands were asked to distribute 20 NCQs each to patients with ≥1 chronic disease. For these patients, continuity is particularly important. At the same time, six medical specialists (oncologist, internist, cardiologist, lung specialist, psychiatrist, and rheumatologist) working in an academic hospital in Nijmegen were asked to distribute 30 questionnaires each to patients in the polyclinic outpatients department.

Patients aged <18 years or those who were unable to speak or read Dutch were excluded. Patients filled in the NCQ at home and could send it back to the researchers. The GP trainees and specialists registered age, sex, and type of chronic disease(s) of participating patients; GP trainees completed some questions on the type of practice in which they worked.

Ethical approval for the study was granted by the ethics committee Arnhem-Nijmegen.

Measurement instruments

The NCQ (Available from the authors on request) consists of 28 items within the following three subscales:

• personal continuity: care provider knows me (five items each for two different providers);

• personal continuity: care provider shows commitment (three items each for two different providers); and

• team/cross-boundary continuity (four items each for three different groups of providers).

Items were scored on a five-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree), with an additional option to choose ‘?’ (‘I do not know’). Patients recruited by the specialist also filled in questions about their GP and vice versa.

As well as the NCQ, all patients were asked to complete the following:

• The ‘care suits patient’ subscale of the Consumer Quality Index General Practice Care questionnaire.12 The questionnaire measures patients’ experiences with general practice care and the subscale measures whether that care suits the patient, corresponding to their experiences with continuity. The subscale consists of nine items, which are scored on a four-point scale ranging from ‘never’ to ‘always’ (Appendix 2 available from the author on request). The option ‘not applicable’ was available for some items. The Consumer Quality Index has shown to be a valid and reliable instrument for measuring the quality of general practice; the higher the score, the better the primary care experience.

• The Continuity of Care from the Client Perspective (VCC) questionnaire.13 This Dutch questionnaire was developed in 1998 to measure continuity of care from the perspective of patients with a chronic disease living at home. The questionnaire comprises items concerning patients’ experiences with their GP, medical specialist, physiotherapist, occupational therapist, dietitian, speech therapist, podiatrist, and home care. Patients were asked to answer only the questions concerning their GP (13 items) and their medical specialist (14 items) (Appendix 3 available from the author on request). Items are scored on a five-point scale, with an additional option of ‘not applicable’. Initial testing has shown that this questionnaire is a valid instrument with medium reliability.

• Two questions on satisfaction (‘I am satisfied with the care I receive from my GP/specialist’).

• Two questions about trust (‘I have trust in my GP/specialist’).

Validity

The following hypotheses were generated to assess construct validity:

1. Principal component analysis would confirm the previously found three continuity subscales11 — personal continuity: care provider knows me, personal continuity: care provider shows commitment, and team/cross-boundary continuity;

2. Higher scores on the three subscales about general practice (personal continuity: GP knows me, personal continuity: GP shows commitment, team/cross-boundary continuity between care providers within general practice) would be highly positively associated with scores on the Consumer Quality Index subscale ‘care suits patient’, the VCC the subscale general practice, and GP trust and satisfaction scores.

3. Scores on the three subscales about hospital/outpatient department care (personal continuity: specialist knows me, personal continuity: specialist shows commitment, team/cross-boundary continuity between care providers within hospital/outpatient department) would be highly positively associated with scores on the VCC subscale specialist care and specialist trust and satisfaction scores.

4. Scores on the team/cross-boundary continuity between GP and specialist subscale would be at least moderately positively associated with scores on the Consumer Quality Index care suits patient subscale, the VCC subscales, and GP and specialist trust and satisfaction scores.

Discriminative ability

The discriminative ability was tested by examining differences in the NCQ subscores for different subgroups. The following hypotheses were generated:

1. Patients recruited from general practice would experience greater continuity in general practice than in hospital/outpatient departments, whereas patients recruited from hospital/outpatient departments would experience greater continuity in hospital/outpatient department. It was expected that, in general, patients recruited from general practice would contact the hospital/outpatient department less frequently than patients recruited from hospital/outpatient departments and vice versa. Contacting a department infrequently was likely to diminish the levels of continuity experienced by the patient.

2. Patients who were psychiatrically ill would experience less continuity than those who were somatically ill, for example, those with diabetes mellitus. This hypothesis is powered from the literature.1,14,15

3. Patients registered in a general practice in a small town (<40 000 inhabitants) would experience more continuity in general practice than patients registered in a general practice in a large town. Patients from a small town are often found to be healthier than their counterparts in larger towns,16 thereby probably seeing fewer care providers, and tending to move less often to another general practice,17 which increases experienced continuity of care.

4. Patients who contacted one provider in the previous year would experience more personal continuity — be that in general practice or in hospital/outpatient departments — than patients who contacted >1 provider.

Reliability

Test–retest reliability was assessed by having participants complete the NCQ a second time, 2 weeks after their initial completion. No intervention took place in these 2 weeks. In the first questionnaire, participants were asked whether they were willing to fill in one more questionnaire after 2 weeks; if so, they had to write down their address so the retest could be sent by mail. No reminder was sent when participants did not respond.

Analyses

SPSS (version 16.0) was used to analyse the data. Item means, total subscale scores, and the percentage of responders with the highest and lowest subscale scores (ceiling and floor effect) were assessed. The subscale scores were calculated as the mean of the items in each subscale. To calculate this score, cases that were missing more than one question within a subscale were excluded.

Confirmatory principal component analysis with varimax rotation was used to verify the three continuity subscales that had been found previously.11 Principal component analysis was performed on the eight items about the patient–provider relationship across the multiple care settings and per care setting separately. In the first analysis, several observations from one patient are included (observations from the provider in general practice and in hospital/outpatient departments); in the second analysis, the observations are all independent. Principal component analysis was also performed on the four items regarding the collaboration and information exchange between the groups of providers across the multiple care settings and per care setting separately.

Validity was determined by examining the correlations between the total scores for all scales using Pearson’s product moment correlations. A moderate correlation was considered to be 0.3 to ≤0.5, and a high correlation was defined as ≥0.5.18,19 To calculate the total score for each scale, cases were excluded if more than one question within each scale was missing.

Independent student t-tests were conducted to determine the discriminative ability. To determine test–retest reliability, the intraclass correlation coefficient (ICC, two-way random effects model, absolute agreement) for the subscale scores was calculated. Reliability was assessed as good with an ICC of >0.70.20 Furthermore, the consistency of measurements was verified using the method described by Bland and Altman.21 The mean difference between the two measurements and the 95% limits of agreement (mean ±1.96 standard deviation of difference) were calculated for each subscale score. The consistency of measurement according to a Bland-Altman plot for one subscale was visualised.

Patient characteristics

Table 1 shows patients’ characteristics and their medical care. Responders and non-responders did not differ in age (P = 0.34). Responders were more likely than non-responders to be male (P = 0.006).

Item and subscale analyses

Table 2 shows item means, total subscale scores, the percentage of patients with the highest and lowest subscale scores, and Cronbach’s alpha for the subscales. The percentage of patients with the highest or lowest subscale score was low (<7.5%), so the NCQ does not show a ceiling or floor effect. Internal consistency (Cronbach’s alpha) ranged from 0.86 to 0.96.

Principal component analysis confirmed the three factors that were found in a previous study (hypothesis one, construct validity);11 it was performed on the eight items about the GP and most important medical specialist together and for each care setting separately. Table 3 shows the factor loadings of this first analysis. It resulted in the same two factors as the last analysis — personal continuity: care provider knows me; personal continuity: care provider shows commitment — explaining 73.8% of the overall variance. A principal component analysis was also performed on the four items about collaboration between the groups of providers across the multiple care settings and per care setting separately. This resulted in the same single factor (team/cross-boundary continuity), explaining 88.8% of total variance.

Construct validity

Table 4 shows the correlations between the NCQ and patient variables. As in hypothesis two, high correlations were found between the three subscales on general practice and the care suits patient subscale of the Consumer Quality Index (r = 0.57–0.75, P<0.01), the general practice subscale of the VCC questionnaire (r = 0.58–0.61, P<0.01), and GP trust (r = 0.59–0.64, P<0.01) and satisfaction (r = 0.63–0.67, P<0.01) scores.

High correlations were also found between the three subscales on hospital/outpatient department care and the specialist care subscale of the VCC questionnaire (r = 0.56–0.73, P<0.01, hypothesis three). Two subscales (personal continuity: specialist knows me; personal continuity: specialist shows commitment) were highly correlated to specialist trust (r = 0.56–0.59, P<0.01) and satisfaction (r = 0.54–0.59, P<0.01) scores, whereas one subscale (team/cross-boundary continuity between care providers within hospital/outpatient departments) was moderately correlated to specialist trust (r = 0.46, P<0.01) and satisfaction (r = 0.48, P<0.01) scores (hypothesis three).

The team/cross-boundary continuity between GP and specialist subscale was at least moderately associated with the care suits patient subscale of the Consumer Quality Index (r = 0.47, P<0.01), VCC subscales (r = 0.56–0.65, P<0.01), GP trust (r = 0.30, P<0.01), GP satisfaction (r = 0.38, P<0.01), and specialist satisfaction (r = 0.33, P<0.01) scores. It was weakly correlated with specialist trust scores (r = 0.27, P<0.01) (hypothesis four).

Discriminative ability

Table 5 shows the continuity subscores of different subgroups. As outlined in hypothesis one, patients recruited from general practice experienced significantly more personal and team continuity in general practice, whereas patients recruited from hospital/outpatient departments experienced more personal continuity in hospital/outpatient departments. The score of team continuity in hospital/outpatient departments did not differ between these subgroups.

In agreement with the second hypothesis, patients who were psychiatrically ill experienced significantly less personal continuity from their GP than those with diabetes. No difference was found in other continuity subscores.

As suggested in hypothesis three, patients registered in a general practice in a small town experienced more personal and team continuity in general practice than patients registered in a general practice in a larger town.

Patients who saw one provider in the previous year experienced more personal continuity in general practice than patients who saw more providers (hypothesis four). No differences in personal continuity in hospital/outpatient departments were found (hypothesis four).

Reliability

In total, 184 patients (69%) agreed to fill in a repeat questionnaire, of whom 172 (93%) returned the retest. Patients who agreed to participate did not differ from those who did not agree in terms of age (P = 0.87), sex (P = 0.47), number of chronic diseases (P = 0.76), and NCQ subscale scores (0.18 to 0.96).

Table 2 shows the ICC per subscale for each provider(s) or group of providers; ICCs varied between 0.71 and 0.82. Table 2 also shows the mean difference between the two measurements and the limits of agreement for each subscale. The mean difference between the two measurements varied between –0.10 and 0.11. The limits of agreement are smallest for the personal continuity: care provider knows me subscale. Figure 1 shows the Bland-Altman plot of the personal continuity: GP knows me subscale. It visually shows the mean difference between test and retest (0.07) with its 95% limits of agreement (–082 to 0.97) plotted against the mean of the sum scores.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Bland-Altman plot of the personal continuity: GP knows me subscale. Intra-individual differences between test and retest responses plotted against the mean of the sum scores. The solid line represents the mean of the intra-individual differences. The dashed lines define the 95% limits of agreement. Mean of the difference ± 1.96 standard deviation.

Summary

This study provides evidence for the validity, discriminative ability, and reliability of the NCQ as a generic questionnaire that measures patients’ experiences of continuity of care as a multidimensional concept, regardless of care setting and morbidity.

Building on previous research,11 this study provides further evidence of its construct validity through the results of the confirmatory principal component analysis and the hypothesised correlations found between the NCQ and other scales measuring quality of care, continuity of care, trust, and satisfaction. Evidence for the discriminative ability of the NCQ is provided by (hypothesised) differences in continuity subscores of different subgroups. The reliability is further supported with the results of the test–retest and the internal consistencies of the subscales.

Strengths and limitations

One limitation is that only patients with a chronic disease were included, which reduces generalisability of the tool. However, patients with ≥1 chronic disease were purposively selected as, for these patients, continuity is particularly important.22–24

Another limitation is the possible recruitment bias. Providers could decide for themselves which patients to ask to participate. GP trainees, more than medical specialists, approached fewer patients than requested. Data of patients that met the inclusion criteria, but were not approached by their provider, were not available.

A last limitation is the finding that, as hypothesised, patients experienced more continuity over the place of recruitment. This may also reflect the likeliness to express satisfaction with the care organisation in which participants are given the questionnaire. The study tried to reduce this potential bias by asking patients to fill in the NCQ at home and send it back to the researchers.

A Cronbach’s alpha of >0.90 was found on the team/cross-boundary subscale, which can imply item redundancy. However, this subscale includes only four items, so was not necessary to shorten the questionnaire.

The NCQ does not show a floor or ceiling effect, so it could perhaps be capable of showing changes in continuity scores over time (responsiveness). This will need further testing.

A strength of this study is that patients from both primary and secondary care were included. The sample size and response rate of participants were high, which strengthens the results and conclusion of this study.

Comparison with existing literature

In a previous study, the NCQ proved to be promising for use with patients in primary care.11 In this preliminary study, the internal consistencies of the subscales and the interscale correlations also provided evidence of a reliable and valid questionnaire with good discriminant abilities.

Implications for practice

Nowadays, an increasing number of providers are involved in the care of patients, especially patients with a chronic disease; this can threaten the continuity of patients’ care. The NCQ is able to identify problems and could evaluate interventions or changes in healthcare systems aimed at improving continuity of care. This will be an important area for further research, given that poor continuity is suspected to have a negative impact on the health of people and populations and also increases medical errors.1–4 Moreover, the NCQ can be used to compare continuity experiences for different diseases and multimorbidity patterns.