Can doctors predict patients' satisfaction and enablement? A cross-sectional observational study

Abstract

Background. Patient satisfaction surveys are increasingly used to measure consulting quality and outcome, but little is known of how good doctors are at judging their patients' satisfaction with their consultations.

Objectives. To determine if patient satisfaction and enablement following a consultation are correlated with both inexperienced and experienced doctors' predictions of patient satisfaction and doctors' own satisfaction with the consultation.

Methods.Design: Cross-sectional questionnaire-based observational study. Setting: Scottish general practices. Main outcome measures: Using a post-consultation questionnaire we compared doctors' estimates of patient satisfaction with the consultation; doctors' own satisfaction with the consultation; scores on the Patient Enablement Instrument (PEI) and the Consultation Satisfaction Questionnaire (CSQ).

Results. Twenty-nine doctors and 1848 patients took part. Each doctor recorded an average of 63.7 (SD: 32.1) consultations. Patient measures of satisfaction and enablement were only weakly correlated with doctor predictions of patient satisfaction (r_s = 0.07 for PEI and 0.13 for CSQ; both P < 0.01) or doctor satisfaction (r_s = 0.10 for PEI and 0.12 for CSQ; both P < 0.01) with the consultation. They were, however, moderately well correlated (r_s = 0.50 P < 0.01) with one another.

Conclusion. Doctors are poor at predicting patient satisfaction in the consultation. Further research is required to determine the reasons for this.

McKinstry B, Colthart I and Walker J. Can doctors predict patients' satisfaction and enablement? A cross-sectional observational study. Family Practice 2006; 23: 240–245.

Introduction

Patient assessment is being increasingly used as an indication of consultation quality. There have been concerns that some doctors might be able to ‘game’ such assessments¹ by selecting patients whom they predict will be satisfied by the consultation, but their ability to do this is unknown. Recent studies have suggested that assessments of consulting skills by researchers using models of consulting skill proposed by influential medical educators agree poorly with patient measures of consultation satisfaction or effectiveness.^2,3 Some doctors have suggested that at least part of this lack of agreement may reflect differences between doctors and patients on the best management of clinical problems (for example, the prescribing of antibiotics⁴ where doctors may be satisfied that they have not prescribed despite recognizing that patients' preferences have not been met).

A number of studies have found that doctors cannot accurately predict their patients' satisfaction with consultations.^5–8 All but one of these studies was very small and the one larger study⁸ of 500 patients and 31 doctors was carried out in a Danish oncology setting. We thought in the UK, where family practitioners often have a long and close relationship with patients, that they may be more successful at predicting their patients' satisfaction.

We attempted to see how well both experienced and inexperienced doctors' estimations of their patients' satisfaction with their consultations agreed with two established and validated patient measures of satisfaction and outcome using the Consultation Satisfaction Questionnaire⁹ (CSQ) and the Patient Enablement Instrument¹⁰ (PEI). A secondary objective related doctors' perceptions of patient satisfaction with consultations to a measure of the doctors' own satisfaction with the same consultations.

Methods

Participants

As a means of obtaining a convenient sample of relatively inexperienced and experienced doctors, all GP registrars in Lothian within 2 months of completion of their training year (n = 62) were invited to take part in the study. Those who volunteered (n = 11; 18%)were also asked to invite principals in their training practices to take part (potentially 65 doctors in 11 practices) but in the end only three training practices decided to take part with 18 doctors (90% of the doctors in the practices, 28% of all potential participants) participating.

Instruments

We used a piloted patient questionnaire which consisted of a two-sided questionnaire containing the CSQ and PEI. The CSQ is a validated 18-question instrument which uses five-point scales to explore consultation quality in four domains: general satisfaction, professional care, depth of relationship and perceived length of consultation. The PEI is a validated measure of consultation outcome and is a six-item instrument which uses a three-point scale centred on the patients' understanding of their health and health problems, and their ability to cope with these as a result of the consultation. Doctors were asked first, to gauge their patients' general satisfaction with their consultation on a four-point scale (‘very’, ‘quite’, ‘not so’ and ‘not at all’ satisfied); and second, their own satisfaction with how they had performed in the consultation on a matching scale. The scale was piloted with a small number of registrars and trainers and minor modifications made.

Data collection

Participating doctors were asked to arrange for 100 consecutive patients (age > 14 years) attending routine surgeries over a period of 2–3 weeks to be given the questionnaire along with an explanatory letter by the receptionist. Patients waiting to see the doctor had the opportunity to read the questionnaire before their consultation, but were asked not to complete it until afterwards. They were asked to give the form to the doctor on entering the consulting room. At the end of the consultation the doctors recorded, on an encrypted part of the form, how satisfied they thought the patients were with the consultation, as well as adding an assessment of the doctor's own satisfaction with the consultation.

The GPs then asked their patients to complete the questionnaire, which they subsequently sealed and placed in a box in reception. Patients were assured in the covering letter that they could not be identified and that their answers would not result in any penalization of the GP.

Analysis

Spearman's rank correlation was used to assess the levels of association between the doctor-recorded measures (of doctors' perception of patients' satisfaction with their consultations and of GPs' own satisfaction with the same consultations) and scores on the patient-completed instruments (PEI and CSQ). Both of the doctor-completed measures were coded 1 = ‘not at all’, 2 = ‘not so’, 3 = ‘quite’ and 4 = ‘very’, higher values thus denoting greater levels of satisfaction. Correlation coefficients were calculated (i) at the level of individual consultations and (ii) at doctor level, with scores on all measures being aggregated as mean values. Intra-class correlation coefficients (ICC) were calculated for all four measures, to determine how much of the total variation in each measure was attributable to higher-level units (i.e. individual doctors). Analyses were carried out for registrars and principals separately, and for both categories aggregated.

Results

Twenty-nine doctors agreed to take part; of which 11 were registrars. All of these doctors were from training practices. The ‘experienced’ doctors had a mean age of 42.0 (SD: 7.1). 75% of these doctors were members of the Royal College of General Practitioners (RCGP) of the UK. The registrars (doctors just completing their general practice training) had a mean age of 27.7 (SD: 1.2). All of these had just sat the examination for membership of the RCGP. In all 1848 consultations were recorded, the average number of consultations recorded per doctor being 63.7 (SD: 32.1 consultations). Of these 1848 consultations, 855 (46.3%) were hosted by registrars, the remaining 993 consultations (53.7%) being with more experienced doctors. The mean age of participating patients was 43.9 years (SD: 17.2 years); 68.2% of patients were female.

Doctors who took part generally found the process easy to manage, although three found it intrusive. Those who gave an explanation for not taking part said that they felt that they did not have the time or that the practice was too busy to participate. Both doctor questionnaires (97%) and patient questionnaires (91% for PEI and 87% for CSQ) were well completed. With 1446 consultations with complete data available for analysis the study had the power to detect as significant (at P = 0.05) values of the Spearman rank correlation coefficient of 0.05 or greater. The view was taken that a correlation coefficient of 0.3 or greater would be of interest, as indicating at least a modest degree of association between doctors' perceptions and patients' reported levels of satisfaction.

Summary measures for responses to the two patient-completed instruments (PEI and CSQ) are shown in Table 1, while the distributions of responses to the doctor-completed items are given in Table 2. Table 3 displays Spearman's rank correlation coefficients for the measures (doctors' perception of patient satisfaction, doctors' own satisfaction, PEI score and CSQ score) for both registrars and more experienced doctors. Correlation coefficients are also shown for both categories of doctor combined.

From this, it is apparent that the patient assessments (the PEI and the CSQ) were moderately well correlated (r_s = 0.50 for registrars and principals combined). This corresponds very closely to the level of association between PEI and CSQ observed in an earlier study.¹¹ However, levels of association between the two doctors' assessments of satisfaction (their own and their perceptions of patient satisfaction) and the PEI and CSQ were weak. In the case of the CSQ this applied both to the total CSQ score and to scores on the four individual domains. Although not shown in Table 3, all correlations between individual CSQ domains and the two doctor assessments of satisfaction were 0.17 or lower.

In 337 of 1848 consultations (18.2%), the doctor's own satisfaction and the patient's perceived satisfaction were discordant; that is, the doctor felt that the patient's assessment of the consultation had been different from the GP's own perception of the consultation. Table 4 shows the cross-tabulation of responses to the two doctor-completed measures, for registrars and principals combined.

Discordant consultations, where doctors perceived their own satisfaction with the consultation to be different from that of the patient, were analysed separately. When doctors perceived that patient satisfaction was less than their own satisfaction (10.1% of all consultations) average PEI and CSQ scores for these patients were very similar to the scores found for these measures when the doctors perceived the patients to be similarly satisfied to themselves (78.4% of all consultations) (Table 5). However in the situation where the doctors were less satisfied by their consultation than they perceived the patients to have been, PEI scores were lower than in the other groups. Differences in PEI score among the three groups were statistically significant (Kruskal–Wallis test: P = 0.01). CSQ scores, however, were broadly similar across all three groups.

ICCs were calculated for all four measures, to determine how much of the total variation in each appeared to be attributable to the effect of higher-level units (i.e. doctors). Values of ICCs—reflecting the respective proportions of total variation which occurred between doctors—were 0.24 (doctor's own satisfaction with consultation); 0.26 (patient's perceived satisfaction with consultation); 0.02 (PEI score) and 0.12 (CSQ score).

Because of the possibility that patients providing incomplete responses might have different views from those who did not, the associations shown in Table 3 were recalculated for these responses. For example, where a patient recorded an incomplete PEI score, the remaining correlations—those not involving PEI—were obtained. Levels of association for these patients were found to be very similar to those reported in Table 3.

When aggregated data at doctor-level were analysed, correlations were similar to those at patient level. Doctors recording fewer than 30 valid scores on the PEI were excluded leaving 23 doctors and 1728 consultations. Aggregation to doctor level was achieved by calculating mean values for the four quantities measured; for the two doctor-completed measures, this process assumed that the variables were measured on interval scales. Table 6 shows rank correlation coefficients for the doctor-level aggregated (mean) values. For individual doctors, only two demonstrated a statistically significant but modest correlation between their assessment of patient satisfaction and the patient-completed measures.

Discussion

Methodological issues

This study was conducted on a convenience sample of general practice trainees and their training practice partners designed to reflect relatively experienced and inexperienced doctors. Although all the trainees in the year were invited, it was not surprising that at the end of a year packed with assessments that a modest number took part. While only a small number of practices agreed to take part, 90% of the partners within those practices volunteered. The low participation rate among doctors approached to take part was not surprising, since the requirements of the study might well have been perceived as burdensome, and from which they would derive little benefit from taking part. Participating doctors were not only required to record additional information during the consultation for the purposes of the study, but they also had to effect a physical exchange of the study questionnaire with the patient. Low participation is a problem which bedevils much consultation research in primary care settings, which is invariably carried out in volunteer populations and there is no easy solution.

However, if anything we expected this volunteer group to be more able than most general practitioners to self analyse their consultations. Consultation analysis takes up a major part of the training year and training practice activity during which subjects had spent many hours reviewing and analysing videos of their and others consultations. This is, we feel, borne out by the fact that our self-selected subjects scored better on enablement than those participating in a larger UK study (mean PEI 4.5, SD 3.9 versus mean PEI 3.1, SD 3.4)¹ suggesting a degree of consulting expertise. It is all the more surprising therefore that they were quite unable to assess their patients' satisfaction. While we realize that it is possible, we think it unlikely that they were likely to be worse than GPs generally.

There was a variable return rate of questionnaires from patients and some of the principals abandoned the study early as they felt it to be too intrusive. Due to the unbalanced nature of the data (i.e. differing numbers of responses from individual doctors), the results must be interpreted with caution. The ICCs suggest that, particularly for the two doctor-completed measures, there is appreciable between-doctor variation in responses. Consequently, the fact that doctors are contributing differentially to the data (i.e. those with greater numbers of responses exert a greater influence on the results than those with fewer responses) limits the inferences which may be drawn. However, even if the analysis is restricted to those returning more than 50 questionnaires the general results remain unchanged. While the simple four-point doctor satisfaction scale was not formally validated, it had good face validity in both piloting and during the trial, was well completed suggesting good acceptability and was internally consistent. While it could be argued that it was not clear which particular component(s) of satisfaction such a simple instrument was measuring it was noteworthy that doctors' perception of patient satisfaction measured this way did not exhibit meaningful levels of association with any of the separate components of the CSQ.

Clinical implications

As others have demonstrated, crucial patient outcomes such as compliance with treatment may be related to patient satisfaction.¹² It is important therefore for doctors to develop ways of recognizing dissatisfaction within the consultation so that they may identify and address timeously patients' concerns.

Overseas researchers in small trials in primary care^5,6 and in critical care,⁷ as well as in a larger trial in oncology,⁸ have all found that doctors are poor at predicting their patients' satisfaction. However given the closer relationship many GPs in the UK have with their patients we had expected UK GPs to be better at predicting how their patients felt about their care.

It is not clear why they are not. Despite research that suggests that doctors and patients agree on the important components of consultations,¹³ it may be that they place different emphasis on the importance of these components. This would certainly be consistent with previous work that shows poor correlation between quality parameters as perceived by doctors and patient satisfaction scores.^2,3 Another possibility is that patients' scoring of consultation satisfaction and outcome, although supposedly relating to a specific consultation, may actually reflect the patients' experience of several consultations.

Possibly doctors judge patients' satisfaction more by the apparent social ease of the consultation than its outcome. In the small number of scenarios where doctors perceived the consultation to have been unsuccessful, but thought that the patient perceived it as successful, PEI (but not CSQ) scores were intriguingly low. We suggest that one possible interpretation of this result is that despite the fact that the consultation had proceeded in an amiable way, the doctor felt that no positive benefit had come from it and so had ranked it as a failure, but perceived that because of the socially successful component of the consultation that the patient had gone away relatively happy. However, while this was mirrored in the patients' CSQ score, the patients could perceive as well as the doctor that there had been no concrete gain from the consultation and that this was reflected in the low PEI.

Implication for quality assessment

Finally, in relation to quality assessment, there have also been concerns that doctors faced with patient assessment may try to ‘game’ the system.¹ One way of doing this would be for doctors to select patients they think will assess them positively. This study suggests that they may find this difficult.

Conclusion

Doctors have traditionally resisted the idea that they might not be good judges of how well or effectively they consult. This pilot study suggests that they may need to think again. Doctors were found to be poor at predicting patient satisfaction. Further research is required to determine the reasons for this. It is likely that satisfied patients are more likely to comply with treatment. It is important therefore that doctors develop ways of recognizing dissatisfaction during the consultation so that they may identify and address patients concerns.

Declaration

Ethical approval: Ethical approval was granted by the Lothian regional ethics committee. The study was supported by NHS Education for Scotland.

Conflict of interest: None declared.

Competing interests: None declared.

We would like to thank the doctors and practices that took part, John Howie and Aziz Sheikh for their help in drafting the paper and Rob Elton for statistical advice. This work was supported by NHS Education for Scotland. BM designed and led the project and wrote the paper. IC ran the project, entered the data and helped write the paper. JW analysed the data and helped write the paper.

References

Author notes

aSenior Researcher Community Health Sciences Edinburgh University, bResearcher NHS Education for Scotland, cResearcher Community Health Sciences (RUHBC) Edinburgh University, Scotland