PENS DOWN, STOP WRITING
Desirable though it might be, there is no such thing as a perfect examination. Concerns about the robustness of both norm-referenced and criterion-referenced high-stakes examinations have been expressed for many years, and proposed solutions include the use of construct-referencing to mitigate some of these difficulties. An examination or assessment can be regarded in much the same way as a laboratory test: subject to false negatives and false positives, and for which a ROC (receiver operating curve) can display the performance of the test under different settings against the standard of proof (the pass mark or the cut-off point). Unsurprisingly, high stakes examinations in medicine — qualifying and licensing examinations — have been subject to a great deal of scrutiny and formal research to determine the presence or absence of systematic error and bias, Equally unsurprisingly they, too, fall short of perfection despite continuous efforts to achieve the highest levels of objective assessment and fairness, and to identify and deal with any suggestion of systemic bias. It is reassuring that two articles in this month’s Journal report carefully conducted studies on selection for training and on the membership examination of the RCGP.
Fiona Patterson and colleagues have examined the predictive validity of selection for postgraduate training in general practice in over 2000 GPs and conclude that their findings ‘... provide good evidence of predictive validity of the selection methods, and the first reported for entry into postgraduate training’, and ‘... show that the best predictor of work performance and training outcomes is a combination of a clinical problem-solving test, a situational judgement test, and a selection centre.’.
In a currently more controversial area, Mei Ling Denney and colleagues report an analysis of over 50 000 candidate cases in the MRCGP clinical skills assessment, with particular reference to the possibility of bias due to ethnicity, sex, or the source of the first medical degree, and despite identifying certain, somewhat counterintuitive and inconsistent, examiner–candidate interactions, they were unable to find evidence of systematic bias.
Good news, too, from Oxford, whence Trevor Lambert and colleagues’ analysis of over 3000 doctors who graduated up to 10 years ago and are now working in general practice revealed high levels of job satisfaction, whether or not general practice had been their original career choice. ‘On this evidence’, they say ‘most doctors who turn to general practice, after preferring another specialty in their early career, are likely to have a satisfying career.’.
This is no time, however, for complacency, because there are still misperceptions about that need to be faced and corrected, a growing undergraduate culture of complaints, appeals, and litigation that tells its own story and definite change — not for the better — in the social and professional fabric of general practice and in the NHS more generally.
Some of the trouble starts in medical school. Many undergraduate cohorts are now so enormous, with getting on for 500 students in each year that it is impossible for teaching staff and pastoral care-givers to do their job properly. The large student body can lead to the formation of cliques based on certain common identities. Enthusiasm for curricular integration has resulted, among other disasters, in the virtual disappearance of the ‘firm’ (and sometimes course) structures in many medical schools, so that students acquire a poor understanding of teamwork, delegation, mutual trust, and leadership. This sorry situation has been worsened immeasurably by the European Working Time Directive, so that some eager students, wishing perhaps to shadow a junior doctor, suddenly find that he or she has finished their shift and is going home at 5 o’clock. I understand that it is now the exception, rather than the rule, for students to take a full part in night-time on-call and emergency activities. Some students will qualify with a very partial knowledge and experience of the social and professional dimensions of medicine, and will be less well equipped for an increasingly challenging career.
