Abstract
Selection into GP speciality training is based on results of a multi-method job analysis study. Six key competency domains were identified as priorities to assess through the current national selection process, including empathy, communication, integrity, clinical expertise, problem-solving, and resilience. Each applicant is assessed using clinical problem-solving and situational judgement machine marked tests, followed by high fidelity exercises at regional selection centres. These show good internal reliability and predictive validity, with high correlations with subsequent job performance and outcomes in the MRCGP examinations. Candidate feedback is generally positive, where candidates prefer multiple opportunities to demonstrate their aptitude. When comparing selection methods, candidates perceive high fidelity assessments (for example, a consultation exercise with a simulated patient) as the most job relevant and fair. Emerging data provides opportunities to both streamline and re-engineer the process, so as to optimise efficiency (including cost-effectiveness) while further developing its robustness. Logistic considerations favour delivery of multiple equated versions of machine marked tests in invigilated test centres. As with other assessments in medical education, concerns remain about differential performance between UK and international medical graduates. There is an urgent need to review the job analysis and selection criteria given profound changes in UK general practice taking place over the last decade.
INTRODUCTION
It seems odd to many GPs, in a specialty that prides itself on patient-centred medicine, that the selection of general practice specialty training (GPST) registrars is based solely on multiple choice question scores and performance in a series of anonymous exercises at a selection centre. In spite of an extensive evidence base for the robustness of the current selection process, a lingering disquiet remains among trainees, GPs, politicians, and patients in the aftermath of the collapse of confidence in MTAS (Medical Training Application Service) of 2007. This paper reviews evidence for the reliability, validity, fairness, and utility of the current process.
Junior doctors robustly criticise1 any selection process that eschews the opportunity to present evidence of undergraduate excellence and experiences working in primary care, or from interesting extracurricular hobbies and interests, during a face-to-face interview with a senior doctor. The wider NHS remains concerned about the apparent high costs of the current GP selection system, compared to conventional application-form-based shortlisting and interview. It remains essential therefore to justify the competency-based methods used for selecting entrants into the first year of specialty training (ST1).
First, in broad terms, the national process used for selecting GPST registrars within the UK is outlined. The strengths and limitations of the methods used are then reviewed, and the evidence base that underpins them. Potential improvements are discussed, with the aim of stimulating debate about further developments in this important area.
THE CURRENT SELECTION PROCESS
Our specialty is unique in having a well-established national selection process for its training programmes, with agreed selection criteria, selection methods, and standards following international standards of best practice.2 A multi-method job analysis study, undertaken some 10?years ago, identified 11 key behavioural competency domains believed to be prerequisites for successful completion of GP specialty training.3,4 At that time, and after extensive consultation, six of these domains were deemed crucial to target at the point of selection: empathy and sensitivity, communication skills, clinical expertise, problem solving, professional integrity, and coping with pressure. The assessment of each now blueprints the current UK selection process.
The selection methodology is delivered in two distinct stages (Figure 1).
Those meeting the entry criteria for UK medical specialty training are ‘shortlisted’ via two machine-marked tests (MMTs)5: a clinical problem-solving paper testing clinical decision making and cognition, and a situational judgement paper, focusing on non-cognitive abilities and various professional attributes, in which applicants are asked to rank order a series of possible solutions to professional dilemmas in work-related scenarios. The key for the situational judgement paper is validated using a group of subject experts. A minimum pass mark is set for the clinical problem-solving paper using a conventional Angoff methodology.6 All doctors failing to exceed this score are rejected at the MMT stage. The score required to progress to the selection centre stage, however, depends on the competition ratio for each applicant's first-choice deanery.
Successful applicants are then invited for further assessment at a selection centre7 in one of their preferred deaneries. Each applicant undertakes three work-related simulations, each timetabled for 30?minutes so as to elicit behavioural evidence in relation to the target selection criteria (for example, empathy, integrity, and so forth). These include a written exercise in which candidates are asked to prioritise a set of impending work-related issues, justifying the order chosen, and two ‘high-fidelity’ role plays: a group discussion where four candidates discuss and resolve a work-related issue, and a simulated patient consultation. The scoring methodology focuses on evaluating the selection criteria (for example, empathy, integrity, problem solving) listed in the person specification, and rather than assigning global scores for each station, using a multi-trait–multimethod approach. Marginally scoring applicants are reviewed formally at the end of each day, in a standardised, moderated review.
How this fits in
Moves towards nationally coordinated selection for all UK medical specialties have catalysed the adoption of current best practice from contexts such as the military and the civil service. General practice has, through careful and incremental development over the last decade, provided a benchmark for selection into postgraduate training. Given current restrictions in the public sector there is a pressing need to control recruitment expenditure. The high costs of remediation of both trainees and established certificate of completion of training holders, and the huge healthcare budgets that most senior UK doctors have responsibility for, make it essential to avoid a ‘penny wise, pound foolish’ approach. Predictive validity studies are informing a utility analysis of the selection system and ongoing evaluation of reliability, validity and fairness are key.
HOW SUCCESSFUL IS THE CURRENT PROCESS?
The success or otherwise of any high-stakes selection process is conventionally considered under the headings of reliability, validity, acceptability, feasibility, and utility (including cost-effectiveness). How does the current process measure up against each of these?
Not unexpectedly, the reliability of the MMTs is good. In 2009 Cronbach's a for each paper was approximately 0.85, well above the internationally recognised standard for similar high-stakes examinations. The standardised written exercise and high-fidelity simulations in the selection centre are also reliable, and are broadly equivalent to similar assessments found in postgraduate medical examinations (Cronbach's α in 2007 was 0.89).8
Can scores at selection predict the subsequent performance of successful trainees? Encouraging early studies on the long-term predictive validity of GP selection centres have consistently shown a high correlation between selection scores and subsequent performance as a trainee 12 months into training, and also high correlations with the Membership of the Royal College of General Practitioners (MRCGP) licensure examinations at the completion of training. The validation results show that the current selection process significantly outperforms results reported for traditional medical interviews.9 Particularly striking are the emerging data from the situational judgement paper aimed at assessing non-cognitive attributes, which seem to correlate with subsequent trainee performance to an extraordinary degree.10 Data on the group discussion exercise, however, suggest that its incremental contribution to the selection centre outcome is marginally less than either the written exercise or patient simulation.
The GP National Recruitment Office (GPNRO) has systematically collected candidate feedback since 2007.11 Evaluation evidence over several years shows that satisfaction with and confidence in the GP selection process is generally high. Initial evaluation of the situational judgement test is more mixed – a test that young doctors seem to find less relevant compared to a clinical knowledge-based test despite increasing evidence of its value and the focus on important non-cognitive attributes.
The delivery of a single, standardised recruitment system on behalf of all UK deaneries is challenging, complicated, and expensive. Administration is supported by a full-time administrative team based at the GPNRO, with collaborative training, quality assurance, and standard setting involving selectors, question writers, role players, and recruitment leads taking place across the UK. The true costs of the process are difficult to measure accurately, but are probably in the region of £400 per applicant. While modest in terms of the investments typically made by private sector companies in employee selection, this stands out in the broader context of the NHS expenditure. This apparent difference is exacerbated by the extensive involvement of senior clinicians in interviews and shortlisting for other specialties; the opportunity costs of which are typically ignored.
It is possible that there may be other measurable positive outcomes in terms of altered candidate behaviour – consequential validity – such as an enhanced engagement with those parts of the foundation curriculum12 assessed in GPST selection, and a positive effect on the subsequent acquisition of the competencies defined in the GP curriculum.13 These also require formal evaluation. Given the current economic climate, demonstrable efficiency is a crucial focus for further developments. A formal economic evaluation is clearly needed.
HOW MIGHT SELECTION BE IMPROVED?
How should recruitment develop from here? Although the UK process, through long and careful development, has achieved much, there is still a long way to go. While the costs of the process may well justify themselves in the long term, there is undoubtedly a pressure to streamline where possible. With long-term predictive validity results now available, it is possible to decipher the exact contribution of each selection instrument in predicting subsequent performance during and at the end of training. In particular, increasing the contribution that the MMTs make to overall scores, in spite of their lower popularity with applicants, provides an opportunity to re-engineer the selection centres to optimise efficiency without compromising the robustness of the overall process. The incremental value of the selection centres remains significant in evaluating behavioural aptitude relating to domains such as empathy and communication.
Research across all occupational groups confirms that selection is not a perfect science,14,15 and for GPs there remain some criteria that cannot be well assessed at selection (such as coping with pressure), or that lend themselves as issues to be dealt with once in training. Arguably, more could be made of the rich information collected during selection to engage in an early, focused dialogue about a trainee's development needs once training commences.
Removing 6000 foundation trainees from acute trusts on a single day to take both MMTs is proving increasingly untenable in the face of competing service demands. Should several equated versions of each paper instead be available on a computer terminal at an invigilated test centre? While this poses technical challenges, such as the production and equating of multiple test forms, it would allow candidates the flexibility of booking their assessment during a test-window of perhaps a week, at a location close to their home, even if that is currently in Pakistan or in New Zealand.
Evidence consistently shows that non-UK-trained doctors perform less well in selection into specialty training and in postgraduate examinations in this country. This finding is consistent with other postgraduate assessment across all specialties.16 While to an extent it may not be surprising that such a doctor might be unfamiliar with British culture, medical practices, and the peculiarities of the NHS, it is incumbent on those delivering such assessments to prove that they remain truly equitable and unbiased.
Although the selection criteria are reviewed regularly, it is now over 10 years since the original competency framework and selection criteria were defined. Do these remain a robust basis for selection, or is it now time to review these in the light of potential extensions to GPST and the changing work of the GP over the intervening years? What are the key competences needed in the generalist of the future? Which of these differentiate the future GP from doctors destined to work in other medical disciplines? The design of assessments to assess each is challenging. How, for example, does one best construct selection methods to assess domains such as leadership potential and strategic thinking? Such questions require urgent resolution. An opportunity now exists for GPs, trainers, and trainees to contribute to the debate on how selection into GPST should move forward from its strong yet still imperfect current position.
Notes
Funding body
This work was supported by funding provided from all United Kingdom postgraduate deaneries through the Committee of GP Educational Directors (COGPED).
Competing interests
Bill Irish is chair of the UK General Practice National Recruitment Office (GPNRO), and works on behalf of all GP directors and deaneries to ensure appropriate selection to GP Specialty Training. Fiona Patterson is scientific adviser to the GPNRO, and advises the department of health and a number of other medical specialities on issues relating to recruitment and workplace psychology.
Discuss this article
Contribute and read comments about this article on the Discussion Forum: http://www.rcgp.org.uk/bjgp-discuss
- Received March 15, 2010.
- Revision received May 4, 2010.
- Accepted June 15, 2010.
- © British Journal of General Practice, 2010.