The 2.1% dilemma: reassessing video consultation in Simulated Consultation Assessment examinations

Chris Jacobs

doi:10.3399/bjgp25X742701

Published eLetters

eLetter submission guidelines

Submit a Response to This Article

Compose eLetter

Title *

Author Information

Contributors

First Name and Middle Initial *

First or given name, e.g. 'Peter'.

Last Name *

Your last, or family, name, e.g. 'MacMoody'.

Email Address *

Your email address, e.g. higgs-boson{at}gmail.com. PLEASE NOTE: your email address will be published.

Occupation *

Your role and/or occupation, e.g. 'Orthopedic Surgeon'.

Affiliation *

Your organization or institution (if applicable), e.g. 'Royal Free Hospital'.

Statement of Competing Interests

Competing interests? *

Yes

Please describe the competing interests

CAPTCHA

This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.

Vertical Tabs

Jump to comment:

Response: Reliability and Standard Setting in the MRCGP SCA
Professor Rich Withnall

Published on: 10 September 2025
MRCGP SCA: “Unrivalled” reliability is the result of an irregular alpha calculation – with serious sequelae
Richard Wakeford

Published on: 10 September 2025
Face validity of the SCA
Roger Neighbour

Published on: 15 July 2025
The 2.1% dilemma: reassessing video consultation in Simulated Consultation Assessment examinations
Peter Lindsay

Published on: 07 July 2025

Published on: (10 September 2025)
Page navigation anchor for Response: Reliability and Standard Setting in the MRCGP SCA
Response: Reliability and Standard Setting in the MRCGP SCA
Professor Rich Withnall, RCGP Chief Examiner, RCGP
We welcome ongoing debate about assessment methodology and appreciate the opportunity to clarify how the Royal College of General Practitioners (RCGP) sets the passing standard for the Simulated Consultation Assessment (SCA). It is correct to note that the SCA reports higher reliability than many comparable postgraduate assessments. This is because reliability has been calculated at the level of domain scores rather than whole stations. That method was intentional: the SCA is built around three domains of general practice, each judged independently, and so domain-level analysis is the most faithful reflection of the assessment’s structure. We acknowledge that this difference in methodology contributes to the reported “unrivalled” reliability, and we view this as a strength of the SCA’s design, not a flaw. We are not comparing our reliability to others; we are setting the standard for entry into general practice.

Pass marks are set using the well-established Borderline Regression Method, which anchors decisions to expert clinical judgements of minimally competent performance. In line with best practice, the College applies an additional adjustment of +1.64 Standard Errors of Measurement to protect patient safety by reducing the risk of false-positive passes. This methodology is widely recognised and accepted by regulators, including the General Medical Council.

It is important to emphasise that the responsibility for setting the p...
Show More

We welcome ongoing debate about assessment methodology and appreciate the opportunity to clarify how the Royal College of General Practitioners (RCGP) sets the passing standard for the Simulated Consultation Assessment (SCA). It is correct to note that the SCA reports higher reliability than many comparable postgraduate assessments. This is because reliability has been calculated at the level of domain scores rather than whole stations. That method was intentional: the SCA is built around three domains of general practice, each judged independently, and so domain-level analysis is the most faithful reflection of the assessment’s structure. We acknowledge that this difference in methodology contributes to the reported “unrivalled” reliability, and we view this as a strength of the SCA’s design, not a flaw. We are not comparing our reliability to others; we are setting the standard for entry into general practice.

Pass marks are set using the well-established Borderline Regression Method, which anchors decisions to expert clinical judgements of minimally competent performance. In line with best practice, the College applies an additional adjustment of +1.64 Standard Errors of Measurement to protect patient safety by reducing the risk of false-positive passes. This methodology is widely recognised and accepted by regulators, including the General Medical Council.

It is important to emphasise that the responsibility for setting the passing standard rests with the RCGP’s examination board, informed by both examiner judgements and psychometric advice. The approach is quality assured, externally scrutinised, and fully aligned with GMC requirements. We recognise that other approaches are possible, and these were considered as part of the regulatory process. However, it is important to stress that while different methods will always produce different numbers, they do not change the fairness of outcomes or the validity of the standard set. There is absolutely no evidence that the RCGP’s chosen approach has resulted in incorrect decisions or unfair outcomes for candidates.

We also note the comments regarding differential attainment. The College has consistently acknowledged the reality of differential performance and works actively with partners across the education system to understand and address it. Research shows that the causes are complex and multifactorial and cannot be attributed to a single methodological choice in assessment design. The RCGP embedded fairness at the heart of the SCA’s design, engaging widely with stakeholders throughout development to ensure the assessment reflects both candidate needs and patient expectations.

In summary, the SCA has been approved against the GMC’s standards and while our statistical methodology differs from some other postgraduate clinical assessments, it is robust, and evidence based. We remain committed to ensuring the assessment is fair, defensible, and continues to command the confidence of patients, the profession, and the public.
Show Less

Competing Interests: None declared.
Published on: (10 September 2025)
Page navigation anchor for MRCGP SCA: “Unrivalled” reliability is the result of an irregular alpha calculation – with serious sequelae
MRCGP SCA: “Unrivalled” reliability is the result of an irregular alpha calculation – with serious sequelae
Richard Wakeford, Psychometric consultant to the CSA 2007-2018 and to the RCA 2020-2021, University of Cambridge
Responding to criticisms of the RCGP’s Simulated Consultation Assessment (SCA), Roger Neighbour says that: “The most important, but not the only, requirements of any exam are that it be valid and reliable. Reliability is concerned with ensuring that the exam can tell good performance from poor and that it can do so consistently. The internationally accepted measures of reliability are Cronbach’s alpha and the standard error of measurement (SEM). The SCA has an average alpha of 0.868 and SEM of 3.84, which are unrivalled in the world of postgraduate clinical exams.”

The SCA’s alpha is indeed unrivalled in its reported level, but this is only because other Royal Colleges do not calculate it in the way as have the College’s psychometricians. Alpha as a reliability indicator is a measure of the internal consistency of a test: how well overall does candidate performance on a set of OSCE stations reflect their likely performance on a typical set of patient presentations in a real-life surgery? It is calculated from the independently-adjudged station scores of, here, the twelve OSCE cases. In general, the more cases, the higher the alpha – and the more representative is the test of real-life performance.
For unclear reasons, the College’s psychometricians have, instead of using the 12 station scores, calculated alpha from the three domain scores on each station (data gathering and diagnosis; clinical management and med...
Show More

Responding to criticisms of the RCGP’s Simulated Consultation Assessment (SCA), Roger Neighbour says that: “The most important, but not the only, requirements of any exam are that it be valid and reliable. Reliability is concerned with ensuring that the exam can tell good performance from poor and that it can do so consistently. The internationally accepted measures of reliability are Cronbach’s alpha and the standard error of measurement (SEM). The SCA has an average alpha of 0.868 and SEM of 3.84, which are unrivalled in the world of postgraduate clinical exams.”

The SCA’s alpha is indeed unrivalled in its reported level, but this is only because other Royal Colleges do not calculate it in the way as have the College’s psychometricians. Alpha as a reliability indicator is a measure of the internal consistency of a test: how well overall does candidate performance on a set of OSCE stations reflect their likely performance on a typical set of patient presentations in a real-life surgery? It is calculated from the independently-adjudged station scores of, here, the twelve OSCE cases. In general, the more cases, the higher the alpha – and the more representative is the test of real-life performance.
For unclear reasons, the College’s psychometricians have, instead of using the 12 station scores, calculated alpha from the three domain scores on each station (data gathering and diagnosis; clinical management and medical complexity; relating to others), a total of 3 x 12 = 36 scores. This calculation is thus based on a statistical assumption that there are 36 independent estimates of candidate ability, when there are actually only 12 samples of performance, separately and independently marked. Since all three domains will be awarded the same score for many candidate cases (from experience in the precursor exams, 30%-40% of all such cases), entering them separately as ‘independent’ ones demonstrably emphasises the inappropriateness of calculating alpha on this basis.

When taxed at a meeting at the RCGP on 23^rd April with the improbably high alpha of 0.868 for a generalist 12-station OSCE covering a wide range of content, the College’s psychometricians defended their calculation as using “disaggregated data” but agreed that when calculated in the normal way, the SCA’s alpha is very similar to that of the predecessor exam (the Clinical Skills Assessment or CSA) at 0.75. This is unsurprising, as the SCA is essentially a buffed-up CSA delivered and marked over the internet, though with only 12 not 13 stations.

Is such apparent statistical pedantry important, beyond the irregular alpha calculation permitting the arrogation of a quite unjustified “unrivalled” quality to the SCA than it merits? Well, the higher the alpha, the smaller the estimate of test measurement error, the SEM. As in the CSA and the RCA (the temporary replacement for the CSA), the SCA adds 1.64 SEMs to its estimated ‘borderline’ score to set the pass mark, countering test measurement imprecision. 1.64 SEMs calculated in this way sets a pass mark about three marks lower than 1.64 SEMs as commonly calculated.

This pass-mark error is associated with three important changes in exam outcomes, apparent in data reported in the College’s Interim Report on the SCA (www.rcgp.org.uk/mrcgp-exams/annual-reports-and-research#sca-interim-report):

a) a large increased first attempt pass rate from the CSA’s – approximately 6% overall – of 3.5% for UK Graduates (UKG) and 8.5% for International Medical Graduates (IMGs);
b) passing a number of candidates who, with a correctly-calculated pass mark, would have failed; and
c) worsened differential attainment (DA) between IMGs and UKGs as measured by relative first attempt fail-rates – 6.2 times under the CSA, in the SCA, 8.5 times

It appears that the change in the pass rates and DA reported by the RCGP for the new exam is primarily due to the inflated reliability and thus reduced SEM as reflected in the (too-small) upward SEM-based adjustment of the pass/fail standard. The same level of DA in the SCA (8.5 times) is reportedly apparent in the following year (www.rcgp.org.uk/getmedia/08d6b55c-481d-4c0b-9223-52f5027ba86f/sca-performance-interim-report-Nov-23-June-24.pdf). A similar order of DA increase would have occurred in the CSA if its pass mark had been reduced.
Show Less

Competing Interests: None declared.
Published on: (15 July 2025)
Page navigation anchor for Face validity of the SCA
Face validity of the SCA
Roger Neighbour, Former Chief Examiner and President, RCGP
Chris Jacobs has noticed that the video-based Simulated Consultation Assessment (SCA) uses a format replicating only a small fraction of real-life practice (https://doi.org/10.3399/bjgp25X742701). This is a surprise to no one, least of all the team who, after unprecedentedly thorough consultation with psychometricians and stakeholders, developed the exam. But his inference – that the SCA should therefore be suspended or abandoned – is flawed, being based on an incomplete understanding of the often conflicting factors that must be balanced when designing a high-stakes OSCE-type assessment.

If cost, resources and practicalities were of no consequence, the gold standard way of assessing a trainee’s readiness to enter the workforce would probably be for upwards of a dozen examiners each to spend a day observing from behind a one-way mirror the candidate’s consultations with scores of patients selected so as to present with problems from across the entire general practice curriculum. Dream on.

The most important, but not the only, requirements of any exam are that it be valid and reliable. Reliability is concerned with ensuring that the exam can tell good performance from poor and that it can do so consistently. The internationally accepted measures of reliability are Cronbach’s alpha and the standard error of measurement (SEM). The SCA has an average alpha of 0.868 and SEM of 3.84...
Show More

Chris Jacobs has noticed that the video-based Simulated Consultation Assessment (SCA) uses a format replicating only a small fraction of real-life practice (https://doi.org/10.3399/bjgp25X742701). This is a surprise to no one, least of all the team who, after unprecedentedly thorough consultation with psychometricians and stakeholders, developed the exam. But his inference – that the SCA should therefore be suspended or abandoned – is flawed, being based on an incomplete understanding of the often conflicting factors that must be balanced when designing a high-stakes OSCE-type assessment.

If cost, resources and practicalities were of no consequence, the gold standard way of assessing a trainee’s readiness to enter the workforce would probably be for upwards of a dozen examiners each to spend a day observing from behind a one-way mirror the candidate’s consultations with scores of patients selected so as to present with problems from across the entire general practice curriculum. Dream on.

The most important, but not the only, requirements of any exam are that it be valid and reliable. Reliability is concerned with ensuring that the exam can tell good performance from poor and that it can do so consistently. The internationally accepted measures of reliability are Cronbach’s alpha and the standard error of measurement (SEM). The SCA has an average alpha of 0.868 and SEM of 3.84, which are unrivalled in the world of postgraduate clinical exams.

Validity is concerned with whether or not the exam assesses things that actually matter in practice (construct validity); whether it does so in ways that plausibly relate to real life (face validity), and whether success or failure correlates with good or poor patient care (predictive validity). While the SCA can be criticised in terms of its face validity, its construct validity is robust, and many years must elapse before its predictive validity can be established. Moreover, the SCA cases are carefully designed to minimise the effects of the constraints imposed by the video format, such as an incomplete view of body language, or awkwardness of eye contact. Good candidates readily overcome these limitations, and can establish effective rapport and emotional insight nevertheless, scoring full marks in the ‘Relating to others’ domain. In my experience tutoring on SCA preparation courses, very few candidates raise Chris Jacobs’ objection, being thoroughly used to the technology in their daily lives and taking the exam’s video format in their stride even if they are unfamiliar with it in real life. If anything, they are more concerned with the verisimilitude of the simulated patients.

Were we to take Dr Jacobs’ criticisms seriously, the options would be either to revert to a previous format of the exam such as the Clinical Skills Assessment (CSA) or to develop an entirely new module based on face-to-face consultations. Neither of these is practicable. A face-to-face exam like the CSA would remain vulnerable to disruption from future pandemics, ‘leaves on the line’, ‘the wrong sort of snow’, public transport strikes and so on. The CSA also had face validity issues of its own, in that real life practice does not include the presence in the consulting room of a silent assessor making notes on a clipboard.

We must also be sensitive to the impact of our exam on candidates’ finances. As well as being much more expensive to mount than the SCA, the CSA required candidates to fund their travel to London and frequently an overnight hotel stay. Its post-Covid successor, the Recorded Consultation Assessment (RCA) where candidates submitted recordings of their own real-life consultations, though apparently stronger on face validity, had serious issues of standardisation threatening its reliability. Many RCA candidates also reported that the task of cherry-picking which consultations to submit came disproportionately to dominate many months of their ST3 experience.

Like all researchers, Chris Jacobs makes the usual plea for further research. I’m sure that, if eventually evidence of the SCA’s predictive validity becomes available, the successors of the present Examination Board will take due account of it. In the meantime, here in the real world, candidates, the profession and the public can have faith in the MRCGP exam as currently configured.
Show Less

Competing Interests: None declared.
Published on: (7 July 2025)
Page navigation anchor for The 2.1% dilemma: reassessing video consultation in Simulated Consultation Assessment examinations
The 2.1% dilemma: reassessing video consultation in Simulated Consultation Assessment examinations
Peter Lindsay, Portfolio GP, .
We should all be grateful to Chris Jacobs¹ for challenging the appropriateness of the SCA as a means of assessing our colleagues moving from specialist trainee to undertaking general practice in the UK. The SCA offers them a challenge over a period of 12 minutes to demonstrate their clinical skills, having been given a few minutes to obtain some written background of the patient(s). Chris Jacobs is entirely correct—what is required is nothing like the other 97.9% of our consultations.

We also seem to be “training to the test”, building training around those “magic” 12 minutes so that practices monitor the trainee’s progress and competence not by clinical and communication skills but by the ability to complete a consultation in 12 minutes which clearly does not represent current practice as approximately a third of consultations last more than 15 minutes and a fifth last more than 20 minutes.^2,3 Explaining such complex matters as the cardiometabolic renal syndrome to our middle-aged patients and considering the significance of adverse childhood experiences on the mental health of our young adults requires longer times, without which we cannot address patient needs and reduce the cycles of deprivation.⁴ We need to assess the progress of our trainees as they learn to share thoughts and feelings rather than just words:

“I don’t take your words
Merely as words...
Show More

We should all be grateful to Chris Jacobs¹ for challenging the appropriateness of the SCA as a means of assessing our colleagues moving from specialist trainee to undertaking general practice in the UK. The SCA offers them a challenge over a period of 12 minutes to demonstrate their clinical skills, having been given a few minutes to obtain some written background of the patient(s). Chris Jacobs is entirely correct—what is required is nothing like the other 97.9% of our consultations.

We also seem to be “training to the test”, building training around those “magic” 12 minutes so that practices monitor the trainee’s progress and competence not by clinical and communication skills but by the ability to complete a consultation in 12 minutes which clearly does not represent current practice as approximately a third of consultations last more than 15 minutes and a fifth last more than 20 minutes.^2,3 Explaining such complex matters as the cardiometabolic renal syndrome to our middle-aged patients and considering the significance of adverse childhood experiences on the mental health of our young adults requires longer times, without which we cannot address patient needs and reduce the cycles of deprivation.⁴ We need to assess the progress of our trainees as they learn to share thoughts and feelings rather than just words:

“I don’t take your words
Merely as words -
Far from it.
I listen
To what makes you talk -
Whatever that is -
And me listen”.

(Shinkichi Takahashi)⁵

At a time when we are trying to increase recruitment and retention, we are making life unnecessarily more difficult for many of our gifted colleagues who are wonderful GP’s but cannot fit it all into the arbitrary 12 minutes. The SCA is a measure of “12-minute skills” rather than the consulting skills needed for the general practice of a rapidly developing NHS. Maybe we should go back to assessing videos of real-life consultations, looking at the content more than the duration?

References

1. Jacobs C. The 2.1% dilemma: reassessing video consultation in Simulated Consultation Assessment examinations. Br J Gen Pract 2025; 75(756): 300. DOI: 10.3399/bjgp25X742701.

2. de Dumast L, Moore P, Snell KI, Marshall T. Trends in clinical workload in UK primary care 2005–2019: a retrospective cohort study. Br J Gen Pract 2024; 74(747): e659-e665. DOI: 10.3399/BJGP.2023.0527.

3. NHS England Digital. Appointments in General Practice, March 2022. 2022. Available at: https://digital.nhs.uk/data-and-information/publications/statistical/appointments-in-general-practice/march-2022.

4. Gopfert A, Deeny SR, Fisher R, Stafford M. Primary care consultation length by deprivation and multimorbidity in England: an observational study using electronic patient records. Br J Gen Pract 2020; 71(704):e185-e192. DOI: 10.3399/bjgp20X714029.

5. Takahashi S. Triumph of the Sparrow: Zen Poems. 2000. Grove Press.
Show Less

Competing Interests: None declared.