Summative assessment: a historical perspective

BACKGROUND

In 1990 the Joint Chairs of the Joint Committee on Postgraduate Training for General Practice (JCPTGP), The Royal College of General Practitioners (RCGP), and the General Medical Services Committee (GMSC) decided that a Statement of Satisfactory Completion of Training should reflect the actual performance of the GP registrar and that a Certificate of Satisfactory Completion of Training would be a national standard of entry into general practice.1 In 1990 it was unusual for a registrar not to receive a Satisfactory Completion Certificate as this could only be triggered by the informed signature of the trainer. Between 1989 and 1992 the proportion not receiving certificates was 0.26%.

The West of Scotland Deanery had an interest and some expertise in assessment. A model for summative assessment was developed which was based on professional judgement. The detail is included in two MD theses.2^,3 The model developed was regulatory, it would assess the competence of doctors for independent practice and protect the public from doctors whose performance is inadequate. This process would have an effect on education as the end stage would drive the learning during the year. A small pilot was carried out in the West of Scotland Deanery commencing in August 1992 and the full pilot carried out for those commencing their GP registrar year in August 1993.

The aim was to have a valid, reliable, and fair system. This had four components: multiple choice (MCQ); assessment of video tape consultations; an audit project; and the trainer's overall judgement. There was significant opposition from trainers, associate advisers and local medical committees. In the full pilot the regional adviser agreed with the trainers that they would be able to stop the new assessment process for any registrar whose performance was identified as being of concern. Seventeen such registrars were identified and 16 of the 17 trainers exercised this power. Subsequent review by a panel of assessors demonstrated concern over the competence of four of the doctors. The subsequent studies found that 5% of registrars did not reach minimal competence. The JCPTGP Summative Assessment Working Party reported in 19934 and a Steering Group from the UK Conference of Regional Advisers was set up to implement their recommendations. This was chaired by Dr John Hasler and the group had good links with the JCPTGP, GMSC, and RCGP. The Steering Group looked at models available for each of the four components suggested by the JCPTGP report.

THE MODEL: MCQ

It had been hoped to use the multiple choice paper of the MRCGP in summative assessment. The West of Scotland Deanery approached the College in 1993 to see if this would be possible and we ran a pilot study together. However, the College refused further use of the question bank and would only permit the use of the MCQ by candidates taking the whole exam. In the absence of an alternative test of factual knowledge this meant that the MRCGP exam, which was a test of excellence, would be compulsory for all perspective GPs. As a result I approached the Royal Australian College of GPs who agreed to sell their question bank. The question bank had reliability data and some adaptation allowed it to be used in a UK context. With time, problem solving questions were added to the test. This test was held in centres throughout the UK four times annually.

CONSULTATION SKILLS

The Steering Group looked at four models: the West of Scotland model which had been developed for the purpose of summative assessment; a formative model5 which had looked at GPs' performance; the Leicester Assessment Package;6 and a proposal from the RCGP examiners. The College model had still to be developed and tested. The educational assessment model had no cut-off point and was felt therefore to not be suitable for detecting minimal competence. The Leicester Assessment Package did not have a cut-off for GP registrars and the West of Scotland model was chosen.

PRACTICAL WORK

The practical work looked at a number of proposals but the only method which had been tested and was therefore accepted was the audit from the West of Scotland package.

TRAINER'S REPORT

A more detailed trainer's report7 was developed in the Oxford Region and this replaced the report used in the West of Scotland pilot.

In preparation for the introduction in 1996 the West of Scotland Deanery carried out training courses throughout the UK and 582 regional assessors supporting the video and audit components of the package were trained. As a result of summative assessment the number of fails increased in the West of Scotland but throughout the UK only 16 out of 6200 registrars were denied their trainer's signature on completion of the vocational year in practice during 1990 to 1995.

Summative assessment was adopted by the JCPTGP on a professionally-led basis on 4 September 1996.8 The system was that developed in the West of Scotland with the Oxford trainer's report. Summative assessment acquired legal status in 1997 after the updating of the Vocational Training Regulations.9

OUTCOMES

One of the concerns of the new system was the absence of failures in some deaneries and over a 5-year period the proportion of failing registrars in the 22 postgraduate deaneries varied from 1.1% to 10.1% despite regular internal and external reliability checks. This variation was in the application of the consulting skills module. There was no significant variation in the deanery scores in the MCQ. This suggested that the consulting skills model was not being applied in a rigorous manner in some deaneries. During the 5-year period 7643 candidates10 undertook the test with 273 (3.6%) unsuccessful. After additional training 1% did not reach the required standard. Video was the most discriminatory component with 124 failing their video submission and 43 failing after additional training. The next highest fail was the trainer's report with 57 failing at the end of the normal training period and 34 after additional training. The discrimination of the other two components, MCQ and audit, would have been much higher without the opportunity to resit and resubmit. From 2000 a simulated surgery11 was accepted as an alternative and a national project marking schedule12 was accepted as an alternative to the written component in 2001. The quality control and the targeting of deaneries with low fails resulted in a much more equitable system and the variation between the deaneries was much less. There was an increasing number of failures with the trainer's report as the trainers gained confidence in its use.

The video component of the MRCGP was accepted for the consultation skills in 2001 and this demonstrated the value of COGPED (Committee of General Practice Education Directors) and the RCGP working together. Failures in the RCGP video were immediately assessed using the summative assessment model. This was arranged within the normal time frame. Mike Pringle recognised the opportunity presented and the numbers taking the single route rose exponentially; and to date 10 669 GP registrars have undertaken the single route since Spring 2001. Roger Neighbour effectively led the examiners in this change.

The National Summative Assessment Board (NSAB) as a sub-committee of COGPED managed the system and this has worked well in the last 5 years with few problems. Over 25 000 doctors have gone through the system and it is interesting, with revalidation on the horizon, that validation for independent general practice is a recent phenomenon. Within the NSAB a number of colleagues have provided leadership: Margareth Atwood, David Percy, John Hasler, Jacky Hayden, Jamie Bahrami, Jamie Bahrami, David Sowden, Steve Field, and Agnes McKnight.

The journey, at times, was difficult but the end result did benefit patient care. The introduction of summative assessment led to lively and sometimes adversarial correspondence in the BJGP13^,14 and in Education for General Practice.15 Concern was expressed regarding the removal of enjoyment from training and there was reported increase in the number of trainers resigning but it was difficult to obtain accurate information.

GP Specialist Training is a welcome development with the nMRCGP. The increased standards for entry to practice will bring additional benefits to patients. Year one has been difficult but the summative assessment tale would suggest that it will take some years before it all settles. All doctors who entered their ST3 (GP registrar) year in August 2007 undertook the nMRCGP. A number of doctors who had already started this component of their training before that date completed summative assessment.

LESSONS LEARNED

Summative assessment of GP trainees was a significant development in the culture of general practice. Its implementation had to overcome many barriers both from within and without the profession. Although it was ultimately a national process the value of one region driving it can probably not be underestimated. This gave the West of Scotland Deanery 15 minutes in the sun!