- Split View
-
Views
-
Cite
Cite
Brian McKinstry, Bruce Guthrie, George Freeman, David Heaney, Is success in postgraduate examinations associated with family practitioners' attitudes or patient perceptions of the quality of their consultations? A cross-sectional study of the MRCGP examination in Great Britain, Family Practice, Volume 22, Issue 6, December 2005, Pages 653–657, https://doi.org/10.1093/fampra/cmi063
- Share Icon Share
Abstract
Background. Postgraduate examinations are ubiquitous in medicine worldwide, but studies to validate them are rare. The Royal College of General Practitioners of the UK, over the years in an evolving format, has offered a membership examination (MRCGP) which it believes acts as a quality marker for those who sit it and also positively influences the development of family practice generally. It is not clear, however, if this process identifies quality markers that patients can perceive.
Objectives. To determine if possession of the MRCGP (a doctor defined measure of doctor quality) is associated with the patient enablement score (a patient based consultation outcome measure) and family practitioners' attitudes to the work of family practice.
Methods. Design: survey using the Patient Enablement Instrument (PEI) with linked survey data on family practitioner (FP) demography and possession of the MRCGP, and FPs' attitudes and beliefs using the Cockburn attitudinal questionnaire. Subjects: 15 534 adult patients attending 154 FP principals. Setting: 50 family practices in the UK. Outcome measures: the association between possession of MRCGP, and PEI and Cockburn scores was assessed using regression analysis controlling for known confounders.
Results. There was no association between PEI score and possession of the MRCGP. Only one scale of the Cockburn attitude questionnaire (the belief that patients should be involved in decision making) was positively associated with possessing the MRCGP.
Conclusion. Any advantage in physician quality conferred by passing the MRCGP exam was not detected in this study. Further research into the predictive validity of postgraduate examinations is required preferably using a wider variety of patient and audit based methods.
McKinstry B, Guthrie B, Freeman G and Heaney D. Is success in postgraduate examinations associated with family practitioners' attitudes or patient perceptions of the quality of their consultations? A cross-sectional study of the MRCGP examination in Great Britain. Family Practice 2005; 22: 653–657.
Introduction
Postgraduate examinations are ubiquitous in medicine worldwide, but studies to validate them are rare.1 Since 1965 the Royal College of General Practitioners in the UK has offered membership of the college by examination (MRCGP) which is believed to serve two functions: firstly that it is a marker of quality for individual family practitioners (FPs) and secondly that it positively influences the development of family practice as a whole irrespective of individual success or failure in the examination.2 Over time, the examination has evolved from a traditional written paper and an oral examination to a modular examination which includes written and multiple choice papers, video or simulated surgery assessment, and oral examination. While there is some published data on the reliability3,4 and content validity5 of parts of the MRCGP, there is less known about its predictive validity in the sense of its associations of success or failure with differences in FPs' beliefs and attitudes, or in patients' experience of their day-to-day practice.
The relationship between the FP and the patient is at the heart of definitions of UK family practice.6 It is created and maintained in the consultation which has been described as “our sole arena, where almost everything we do as doctors is done”.7 There is a considerable literature on consultation skills, which has been significantly incorporated in the MRCGP and the UK Summative Assessment examinations' video components.2,8 However, an alternative conceptualisation of a FP's ‘performance’ in the consultation is patient assessment rather than external observation.
If the MRCGP is a marker of excellence, then it might be expected that that doctors with MRCGP (particularly more recently) would share the kinds of attitudes identified as good by the RCGP, and would perform better in the consultation than those without. This paper therefore reports an analysis of a large UK dataset examining the association between possession of MRCGP and both FPs' attitudes to their work using the Cockburn scale,9 and an aggregate score of patients' assessment of the outcome of consultation with FPs using the Patient Enablement Instrument (PEI).10
Methods
The dataset used was a survey of patients aged 12 and over attending FPs in four UK regions in a two week period in 1998.
The original sample
50 doctors in 10 practices in each of four participating regions (Lothian, Coventry, Oxfordshire, and west London) were recruited by inviting a random sample of about twice the necessary size to attend a series of briefing meetings. In practices of up to three partners all had to agree to take part, but three out of four partners, four of five partners, or all but two partners in larger practices were regarded as sufficient. The study achieved a 38% take-up rate from the random sample of practices, (higher for larger practices). The practices that declined had a similar demographic profile to those that accepted, but no comment could be made about whether the patterns of care they offered would also have been similar. A cross section of small, large, deprived, and non-deprived practices with a differing ethnic mix of both patients and doctors across the four participating areas was achieved. There was a higher proportion of training practices (40%) in the uptake group than is found nationally, but training practice status was not found to have a significant effect on enablement in the study.
Practices were asked to collect consultation data for two consecutive weeks during March and April 1998. Patients completed a pre-consultation questionnaire (which included questions about demography, reason for consultation, and how well they knew the FP being seen), and a post-consultation questionnaire which included the Patient Enablement Instrument, a patient centred measure of consultation outcome. This study uses the same definitions of case-mix (the problems the patient had or wished to discuss) and PEI scoring as the original study report.10
FPs also completed a questionnaire which included demographic information, how long they had been in this practice, whether they had MRCGP and a Cockburn attitude questionnaire. The Cockburn questionnaire was rigorously developed in Australia to measure FP attitudes,9 and consists of seven scales, six of which map well to accepted good practice in the UK. The seventh scale relating to FP attitudes to the role of the state in medical care does not map well to the UK, and was ignored.
The analysis was restricted to fully trained FPs (principals) since FPs in training (registrars) have not had the opportunity to take MRCGP, and because many locums did not return questionnaires. While we did not have data on when the FPs had sat the MRCGP we assumed that age was a reasonable proxy for this in our model. The dataset analysed therefore consists of Cockburn questionnaire responses from 150 FP principals, and PEI scores from 15 534 adult patients attending 154 FP principals in 50 practices.
Univariate differences between FPs with and without MRCGP in mean attitude and mean PEI scores were analysed using t-tests, with a Bonferroni correction used to account for multiple testing of attitude scores. Both attitudes and behaviour in the consultation are plausibly related to FP age, sex and postgraduate experience, and the PEI score has been shown to be associated with a range of patient, FP and practice characteristics.10 Regression models incorporating plausible confounders were therefore constructed, and the effect of the FP having MRCGP after adjustment for these variables estimated by forcing whether or not the FP had MRCGP into the model in the final step. For the attitude scores, single level linear regression was used, and only FP level variables examined. For the PEI score, because patient, FP and practice characteristics may affect consultation outcome, multilevel linear regression was used, fitting a three level hierarchical model of patients within FP within practice.11 Inclusion of variables in the final model was decided using the deviance statistic to assess model fit, and model assumptions examined using graphical plots. Data was analysed using SPSS version 11 for the univariate analyses and single level regressions,12 and MLwin version 1.1 for the multilevel regression.13
Results
We found no associations between attitude scores and the potential FP level confounders of FP age, sex and time in practice. Hence Table 1 shows the univariate results for the relationship between FPs having MRCGP and attitudes measured by the Cockburn scale. FPs with and without MRCGP significantly differed on only one scale, with those with MRCGP more likely to believe that patients should be involved in decision making. There were no statistically significant differences for other attitudes, and no clear trends favouring one group or the other.
Mean attitude scorea . | Not MRCGP (n = 96 FPs) . | MRCGP (n = 54 FPs) . | Mean difference (95% CI) P-valueb . |
---|---|---|---|
FP has psychological orientation | 4.80 | 5.06 | 0.26 (−0.58 to 0.06) P = 0.116 |
Prevention an important aspect of FPs' work | 5.21 | 5.06 | −0.15 (−0.51 to 0.21) P = 0.414 |
Patients should be equal and active participants in the consultation | 5.90 | 5.67 | −0.23 (−0.50 to 0.04) P = 0.094 |
FPs should be open, listening and provide adequate information | 5.99 | 5.78 | −0.21 (−0.52 to 0.10) P = 0.585 |
Patients should be involved in decision making about treatment | 4.29 | 4.99 | 0.71 (−0.30 to 1.11) P = 0.001 |
Patients rarely consult inappropriately | 3.78 | 4.12 | 0.34 (−0.12 to 0.80) P = 0.142 |
Mean attitude scorea . | Not MRCGP (n = 96 FPs) . | MRCGP (n = 54 FPs) . | Mean difference (95% CI) P-valueb . |
---|---|---|---|
FP has psychological orientation | 4.80 | 5.06 | 0.26 (−0.58 to 0.06) P = 0.116 |
Prevention an important aspect of FPs' work | 5.21 | 5.06 | −0.15 (−0.51 to 0.21) P = 0.414 |
Patients should be equal and active participants in the consultation | 5.90 | 5.67 | −0.23 (−0.50 to 0.04) P = 0.094 |
FPs should be open, listening and provide adequate information | 5.99 | 5.78 | −0.21 (−0.52 to 0.10) P = 0.585 |
Patients should be involved in decision making about treatment | 4.29 | 4.99 | 0.71 (−0.30 to 1.11) P = 0.001 |
Patients rarely consult inappropriately | 3.78 | 4.12 | 0.34 (−0.12 to 0.80) P = 0.142 |
Mean patient enablement scorec . | Not MRCGP (n = 98) . | MRCGP (n = 56) . | Mean difference (95% CI) P-value . |
---|---|---|---|
Crude | 3.28 | 3.22 | −0.06 (−0.19 to 0.32) P = 0.620 |
Adjusted | 2 | 2 | 0.03 (−0.18 to 0.24) P = 0.387 |
Mean patient enablement scorec . | Not MRCGP (n = 98) . | MRCGP (n = 56) . | Mean difference (95% CI) P-value . |
---|---|---|---|
Crude | 3.28 | 3.22 | −0.06 (−0.19 to 0.32) P = 0.620 |
Adjusted | 2 | 2 | 0.03 (−0.18 to 0.24) P = 0.387 |
Each is scored on a range of 1–7, with a higher score representing closer orientation to the statement.
Bonferroni corrected significance level = 0.008.
Adjusted for patient age, patient sex, case mix, whether the consultation was interrupted, whether the patient knew the FP well, whether the patient had 2 or more problems to discuss, whether the patient spoke English at home, FP age and FP sex. Other variables examined, but not included in the final model were whether the patient was seen in an open or booked surgery, whether they were an ‘extra’, whether the FP was vocationally trained, and practice list size.
Mean attitude scorea . | Not MRCGP (n = 96 FPs) . | MRCGP (n = 54 FPs) . | Mean difference (95% CI) P-valueb . |
---|---|---|---|
FP has psychological orientation | 4.80 | 5.06 | 0.26 (−0.58 to 0.06) P = 0.116 |
Prevention an important aspect of FPs' work | 5.21 | 5.06 | −0.15 (−0.51 to 0.21) P = 0.414 |
Patients should be equal and active participants in the consultation | 5.90 | 5.67 | −0.23 (−0.50 to 0.04) P = 0.094 |
FPs should be open, listening and provide adequate information | 5.99 | 5.78 | −0.21 (−0.52 to 0.10) P = 0.585 |
Patients should be involved in decision making about treatment | 4.29 | 4.99 | 0.71 (−0.30 to 1.11) P = 0.001 |
Patients rarely consult inappropriately | 3.78 | 4.12 | 0.34 (−0.12 to 0.80) P = 0.142 |
Mean attitude scorea . | Not MRCGP (n = 96 FPs) . | MRCGP (n = 54 FPs) . | Mean difference (95% CI) P-valueb . |
---|---|---|---|
FP has psychological orientation | 4.80 | 5.06 | 0.26 (−0.58 to 0.06) P = 0.116 |
Prevention an important aspect of FPs' work | 5.21 | 5.06 | −0.15 (−0.51 to 0.21) P = 0.414 |
Patients should be equal and active participants in the consultation | 5.90 | 5.67 | −0.23 (−0.50 to 0.04) P = 0.094 |
FPs should be open, listening and provide adequate information | 5.99 | 5.78 | −0.21 (−0.52 to 0.10) P = 0.585 |
Patients should be involved in decision making about treatment | 4.29 | 4.99 | 0.71 (−0.30 to 1.11) P = 0.001 |
Patients rarely consult inappropriately | 3.78 | 4.12 | 0.34 (−0.12 to 0.80) P = 0.142 |
Mean patient enablement scorec . | Not MRCGP (n = 98) . | MRCGP (n = 56) . | Mean difference (95% CI) P-value . |
---|---|---|---|
Crude | 3.28 | 3.22 | −0.06 (−0.19 to 0.32) P = 0.620 |
Adjusted | 2 | 2 | 0.03 (−0.18 to 0.24) P = 0.387 |
Mean patient enablement scorec . | Not MRCGP (n = 98) . | MRCGP (n = 56) . | Mean difference (95% CI) P-value . |
---|---|---|---|
Crude | 3.28 | 3.22 | −0.06 (−0.19 to 0.32) P = 0.620 |
Adjusted | 2 | 2 | 0.03 (−0.18 to 0.24) P = 0.387 |
Each is scored on a range of 1–7, with a higher score representing closer orientation to the statement.
Bonferroni corrected significance level = 0.008.
Adjusted for patient age, patient sex, case mix, whether the consultation was interrupted, whether the patient knew the FP well, whether the patient had 2 or more problems to discuss, whether the patient spoke English at home, FP age and FP sex. Other variables examined, but not included in the final model were whether the patient was seen in an open or booked surgery, whether they were an ‘extra’, whether the FP was vocationally trained, and practice list size.
GPs without MRCGP had slightly higher unadjusted mean PEI scores, although this was reversed after adjustment in the multilevel regression model. However, the absolute differences were very small, neither difference was statistically significant and the confidence intervals were wide (Table 1).
Discussion
The results show little association between possession of the MRCGP and either measured physician attitudes or patient assessment of consultation outcome as measured by the PEI. The only significant association was that FPs with MRCGP were more likely to believe that patients should be involved in decision making about treatment. These findings persisted even after allowing for likely confounders.
The study has several important limitations, although these do not invalidate the findings. First, although a random sample of practices was approached, participation involved extra administrative work over a two week period and only 53% participated.
Secondly, no FP in this study had sat the current MRCGP video component as the study predated its introduction. It is plausible that the more modern MRCGP examination would be better at identifying FPs with better consultation and communication skills and performance. However, it is known that there is significant overlap in the domains of competence measured by different components of this examination5 and a previous small study examining MRCGP video performance and patient assessment of registrars did not find any association with the exception of one parameter (seeking to confirm patient understanding).14 In addition a descriptive study on patient centredness15 in video submissions for the more recent MRCGP version has demonstrated that even successful candidates often fail to take into account patients' beliefs, confirm their understanding or share management options. All are skills which more plausibly result in a higher enablement score, than the kinds of information seeking consultation skills which those sitting the MRCGP more commonly demonstrate.
Thirdly, the Cockburn questionnaire was developed in Australia and may not properly apply in a different country, although its development was rigorous9 and the scales used in this analysis have strong face validity in the context of UK family practice.
Fourthly, patients' perception of quality may differ from that of doctors. In a retrospective anonymous questionnaire study, in which trainers were invited to give a confidential report on their most recent registrar, trainers perceived that registrars who achieved the MRCGP were superior across a wide range of attributes and were significantly more likely to say they would have them as partners or their own doctor than those registrars who had not.16 However, the centrality of the consultation in FP core values makes it hard to argue that patient assessments of consultation outcomes like the PEI are not important, and the PEI is an approved measure of ‘communication’ for doctors' revalidation.17 Equally though, this study is potentially limited by only using one such measure, and others may be judged more relevant or better for this purpose.
Finally it may be that the main effect of the MRCGP has been to raise the overall standard of family practice. There is some evidence to support this; for example the introduction of a critical reading component to the examination led to a change in reading habits of registrars away from undergraduate textbooks and tabloid medical press towards the BMJ and BJGP.18 It is therefore plausible that the examination has influenced the attitudes and lifted the consultation quality of all doctors in the UK as part of a wider internalisation of its embedded values. Such a process may continue beyond any period of formal training as part of growing experience and continuing professional development, but this hypothesis would be difficult to test.
Measuring performance in practice is difficult.19 The ability to demonstrate competence in examinations is no guarantee that such competencies will be put into practice in day to day work. However, it is reasonable to assume that those who cannot demonstrate such competencies will be unable to put them into practice. There is some evidence from the United States that a relationship between results in certifying examinations and patient outcomes can be made.20 It is also common for UK researchers to include possession of the MRCGP as a possible confounding variable when analysing data from primary care in the belief that this factor may be a positive influence on doctor behaviour. In this context, the lack of association between possession of MRCGP and all but one of the attitudinal measures in this study is striking.
Postgraduate examinations are widely used internationally as markers of excellence, but if they are to have any meaning, then relationships between exam performance and actual clinical practice should be demonstrated. Currently there is no agreement about what measures of actual clinical practice would be expected to be associated with exam performance,21 although the requirements for General Medical Council revalidation,17 RCGP Quality Practice Award22 and the new GMS contract all offer comprehensive measures of good practice. Examining the relationships between these and exam performance will require a mixture of information sources including patient and peer assessment and audit data.19 Subject to appropriate consent and data protection considerations, the rapid computerisation of practice in the UK and the requirements of revalidation should help create the kinds of large datasets necessary for such studies.
Declaration
Funding: Brian McKinstry and Bruce Guthrie are funded by the Chief Scientist Office: Scottish Executive.
Ethical approval: granted for the original study and was not considered necessary for the re-analysis as this involved anonymised data.
Conflicts of interest: none.
References
Hutchison L, Aitken P, Hayes T. Are medical postgraduate certification processes valid? A systematic review of the published evidence.
RCGP. The MRCGP Examination. http://www.rcgp.org.uk/exam/index.asp (accessed 7 December
Wass V, Wakeford R, Neighbour R, Van Der Vleuten C. Achieving acceptable reliability in oral examinations: An analysis of the Royal College of General Practitioners membership examination's oral component.
RCGP. Examination for Membership (MRCGP) Regulations for 2004. http://www.rcgp.org.uk/exam/regulations/2004/regu5.asp (accessed 16 November
Munro N, Rughani A, Foulkes J, Wilson A, Neighbour R. Assessing validity in written tests of general practice—Exploration by factor analysis of candidate response patterns to Paper 1 of the MRCGP examination.
RCGP, General Practitioners' Committee of the BMA, and NHS Alliance. Valuing General Practice. London: RCGP;
Norell J. Uses and abuses of the consultation. In Elder A, Samuel O (eds). While I'm here doctor. London: Tavistock;
National Office for Summative Assessment for General Practice. http://www.nosa.org.uk. (accessed 30 September
Cockburn J, Killer D, Campbell E, Sanson-Fisher RW. Measuring general practitioners' attitudes towards medical care.
Howie JG, Heaney DJ, Maxwell M, Walker JJ, Freeman GK, Rai H. Quality at general practice consultations: cross sectional survey.
McKinstry B, Walker J, Blaney D, Heaney D, Begg D. Do patients and expert doctors agree on the assessment of consultation skills? A comparison of two patients consultation assessment scales with the video component of the MRCGP.
Campion P, Foulkes J, Neighbour R, Tate P. Patient-centredness in the MRCGP video examination: analysis of a large cohort.
McKinstry B, Colthart I, Taylor G, Elliott K. Fitness for purpose among recently qualified general practitioners: The use of a confidential trainer report.
RCGP Scotland. Revalidation folder. Doctors working in clinical general practice. http://www.rcgp-scotland.org.uk/development/docs/Revalidation_Booklet.pdf (accessed 7 December
Wakeford R, Southgate L. Postgraduate medical education: modifying trainees' study approaches by changing the examination.
Rethans J-J, Norcini JJ, Baron-Maldonado M, Blackmore D, Jolly BC, LaDuca T, Lew S, Page GG, Southgate LH. The relationship between competence and performance: Implications for assessing practice performance.
Norcini JJ. Lipner RS. Kimball HR. Certifying examination performance and patient outcomes following acute myocardial infarction.
Campion P. Can you compare competence and performance? A reply to McKinstry et al.
RCGP. The Quality Practice Award. http://www.rcgp.org.uk/faculties/scotcounc/gpa.asp (accessed 7 December
Author notes
aChief Scientist Office Career Fellow, University of Edinburgh, bHealth Foundation/Chief Scientist Office Postdoctoral Research Fellow, University of Dundee, cProfessor Emeritus of General Practice, Imperial College London and dSenior Researcher, University of Aberdeen. Email: brian.mckinstry@ed.ac.uk