Skip to main content

Main menu

  • HOME
  • ONLINE FIRST
  • CURRENT ISSUE
  • ALL ISSUES
  • AUTHORS & REVIEWERS
  • SUBSCRIBE
  • RESOURCES
    • About BJGP
    • Conference
    • Advertising
    • BJGP Life
    • eLetters
    • Librarian information
    • Alerts
    • Resilience
    • Video
    • Audio
    • COVID-19 Clinical Solutions
  • RCGP
    • BJGP for RCGP members
    • BJGP Open
    • RCGP eLearning
    • InnovAIT Journal
    • Jobs and careers
    • RCGP e-Portfolio

User menu

  • Subscriptions
  • Alerts
  • Log in

Search

  • Advanced search
British Journal of General Practice
Advertisement
  • RCGP
    • BJGP for RCGP members
    • BJGP Open
    • RCGP eLearning
    • InnovAIT Journal
    • Jobs and careers
    • RCGP e-Portfolio
  • Subscriptions
  • Alerts
  • Log in
  • Follow bjgp on Twitter
  • Visit bjgp on Facebook
  • Blog
  • Listen to BJGP podcast
Advertisement
British Journal of General Practice

Advanced Search

  • HOME
  • ONLINE FIRST
  • CURRENT ISSUE
  • ALL ISSUES
  • AUTHORS & REVIEWERS
  • SUBSCRIBE
  • RESOURCES
    • About BJGP
    • Conference
    • Advertising
    • BJGP Life
    • eLetters
    • Librarian information
    • Alerts
    • Resilience
    • Video
    • Audio
    • COVID-19 Clinical Solutions
The Back Pages

Summative assessment: a historical perspective

Stuart Murray
British Journal of General Practice 2008; 58 (557): 894-895. DOI: https://doi.org/10.3399/bjgp08X376357
Stuart Murray
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Article
  • Info
  • eLetters
  • PDF
Loading

BACKGROUND

In 1990 the Joint Chairs of the Joint Committee on Postgraduate Training for General Practice (JCPTGP), The Royal College of General Practitioners (RCGP), and the General Medical Services Committee (GMSC) decided that a Statement of Satisfactory Completion of Training should reflect the actual performance of the GP registrar and that a Certificate of Satisfactory Completion of Training would be a national standard of entry into general practice.1 In 1990 it was unusual for a registrar not to receive a Satisfactory Completion Certificate as this could only be triggered by the informed signature of the trainer. Between 1989 and 1992 the proportion not receiving certificates was 0.26%.

The West of Scotland Deanery had an interest and some expertise in assessment. A model for summative assessment was developed which was based on professional judgement. The detail is included in two MD theses.2,3 The model developed was regulatory, it would assess the competence of doctors for independent practice and protect the public from doctors whose performance is inadequate. This process would have an effect on education as the end stage would drive the learning during the year. A small pilot was carried out in the West of Scotland Deanery commencing in August 1992 and the full pilot carried out for those commencing their GP registrar year in August 1993.

The aim was to have a valid, reliable, and fair system. This had four components: multiple choice (MCQ); assessment of video tape consultations; an audit project; and the trainer's overall judgement. There was significant opposition from trainers, associate advisers and local medical committees. In the full pilot the regional adviser agreed with the trainers that they would be able to stop the new assessment process for any registrar whose performance was identified as being of concern. Seventeen such registrars were identified and 16 of the 17 trainers exercised this power. Subsequent review by a panel of assessors demonstrated concern over the competence of four of the doctors. The subsequent studies found that 5% of registrars did not reach minimal competence. The JCPTGP Summative Assessment Working Party reported in 19934 and a Steering Group from the UK Conference of Regional Advisers was set up to implement their recommendations. This was chaired by Dr John Hasler and the group had good links with the JCPTGP, GMSC, and RCGP. The Steering Group looked at models available for each of the four components suggested by the JCPTGP report.

THE MODEL: MCQ

It had been hoped to use the multiple choice paper of the MRCGP in summative assessment. The West of Scotland Deanery approached the College in 1993 to see if this would be possible and we ran a pilot study together. However, the College refused further use of the question bank and would only permit the use of the MCQ by candidates taking the whole exam. In the absence of an alternative test of factual knowledge this meant that the MRCGP exam, which was a test of excellence, would be compulsory for all perspective GPs. As a result I approached the Royal Australian College of GPs who agreed to sell their question bank. The question bank had reliability data and some adaptation allowed it to be used in a UK context. With time, problem solving questions were added to the test. This test was held in centres throughout the UK four times annually.

CONSULTATION SKILLS

The Steering Group looked at four models: the West of Scotland model which had been developed for the purpose of summative assessment; a formative model5 which had looked at GPs' performance; the Leicester Assessment Package;6 and a proposal from the RCGP examiners. The College model had still to be developed and tested. The educational assessment model had no cut-off point and was felt therefore to not be suitable for detecting minimal competence. The Leicester Assessment Package did not have a cut-off for GP registrars and the West of Scotland model was chosen.

PRACTICAL WORK

The practical work looked at a number of proposals but the only method which had been tested and was therefore accepted was the audit from the West of Scotland package.

TRAINER'S REPORT

A more detailed trainer's report7 was developed in the Oxford Region and this replaced the report used in the West of Scotland pilot.

In preparation for the introduction in 1996 the West of Scotland Deanery carried out training courses throughout the UK and 582 regional assessors supporting the video and audit components of the package were trained. As a result of summative assessment the number of fails increased in the West of Scotland but throughout the UK only 16 out of 6200 registrars were denied their trainer's signature on completion of the vocational year in practice during 1990 to 1995.

Summative assessment was adopted by the JCPTGP on a professionally-led basis on 4 September 1996.8 The system was that developed in the West of Scotland with the Oxford trainer's report. Summative assessment acquired legal status in 1997 after the updating of the Vocational Training Regulations.9

OUTCOMES

One of the concerns of the new system was the absence of failures in some deaneries and over a 5-year period the proportion of failing registrars in the 22 postgraduate deaneries varied from 1.1% to 10.1% despite regular internal and external reliability checks. This variation was in the application of the consulting skills module. There was no significant variation in the deanery scores in the MCQ. This suggested that the consulting skills model was not being applied in a rigorous manner in some deaneries. During the 5-year period 7643 candidates10 undertook the test with 273 (3.6%) unsuccessful. After additional training 1% did not reach the required standard. Video was the most discriminatory component with 124 failing their video submission and 43 failing after additional training. The next highest fail was the trainer's report with 57 failing at the end of the normal training period and 34 after additional training. The discrimination of the other two components, MCQ and audit, would have been much higher without the opportunity to resit and resubmit. From 2000 a simulated surgery11 was accepted as an alternative and a national project marking schedule12 was accepted as an alternative to the written component in 2001. The quality control and the targeting of deaneries with low fails resulted in a much more equitable system and the variation between the deaneries was much less. There was an increasing number of failures with the trainer's report as the trainers gained confidence in its use.

The video component of the MRCGP was accepted for the consultation skills in 2001 and this demonstrated the value of COGPED (Committee of General Practice Education Directors) and the RCGP working together. Failures in the RCGP video were immediately assessed using the summative assessment model. This was arranged within the normal time frame. Mike Pringle recognised the opportunity presented and the numbers taking the single route rose exponentially; and to date 10 669 GP registrars have undertaken the single route since Spring 2001. Roger Neighbour effectively led the examiners in this change.

The National Summative Assessment Board (NSAB) as a sub-committee of COGPED managed the system and this has worked well in the last 5 years with few problems. Over 25 000 doctors have gone through the system and it is interesting, with revalidation on the horizon, that validation for independent general practice is a recent phenomenon. Within the NSAB a number of colleagues have provided leadership: Margareth Atwood, David Percy, John Hasler, Jacky Hayden, Jamie Bahrami, Jamie Bahrami, David Sowden, Steve Field, and Agnes McKnight.

The journey, at times, was difficult but the end result did benefit patient care. The introduction of summative assessment led to lively and sometimes adversarial correspondence in the BJGP13,14 and in Education for General Practice.15 Concern was expressed regarding the removal of enjoyment from training and there was reported increase in the number of trainers resigning but it was difficult to obtain accurate information.

GP Specialist Training is a welcome development with the nMRCGP. The increased standards for entry to practice will bring additional benefits to patients. Year one has been difficult but the summative assessment tale would suggest that it will take some years before it all settles. All doctors who entered their ST3 (GP registrar) year in August 2007 undertook the nMRCGP. A number of doctors who had already started this component of their training before that date completed summative assessment.

LESSONS LEARNED

Summative assessment of GP trainees was a significant development in the culture of general practice. Its implementation had to overcome many barriers both from within and without the profession. Although it was ultimately a national process the value of one region driving it can probably not be underestimated. This gave the West of Scotland Deanery 15 minutes in the sun!

  • © British Journal of General Practice, 2008.

REFERENCES

  1. ↵
    1. Irvine DH,
    2. Pereira Gray DJ,
    3. Bogle IG
    (1990) Vocational training for General Practice: The meaning of satisfactory completion [letter]. Br J Gen Pract 40:434.
    OpenUrlFREE Full Text
  2. ↵
    1. Campbell LM
    (1996) The development of a summative assessment for vocational trainees in general practice. MD Thesis (University of Glasgow, Glasgow).
  3. ↵
    1. Lough JRM
    (2002) The development of integrated audit for the training of general practice registrars. MD Thesis (University of Glasgow, Glasgow).
  4. ↵
    1. Joint Committee on Postgraduate Training for General Practice
    (1993) Report of Summative Assessment Working Party (JCPTGP, London).
  5. ↵
    1. Cox J,
    2. Mulholland H
    (1993) An instrument for assessment of video tapes on general practitioners' performance. BMJ 306:1043–1046.
    OpenUrlAbstract/FREE Full Text
  6. ↵
    1. Fraser RC,
    2. McKinley RK,
    3. Mulholland H
    (1994) Consultation competence in general practice: testing the reliability of the Leicester assessment package. Br J Gen Pract 44:293–296.
    OpenUrlAbstract/FREE Full Text
  7. ↵
    1. Johnstone N,
    2. Hasler J,
    3. Toby J,
    4. Grant J
    (1996) Consensus of minimum standards for use in a trainer's report for summative assessment in general practice. Br J Gen Pract 46:140–144.
    OpenUrlAbstract/FREE Full Text
  8. ↵
    1. Carnal D
    (1996) Summative assessment in general practice [editorial]. BMJ 313:638–639.
    OpenUrlFREE Full Text
  9. ↵
    1. National Health Service (Vocational Training for General Medical Practice)
    (1997) Regulation 9(2). Statutory Instrument 3150 (HMSO, London).
  10. ↵
    1. Stuart MT,
    2. Atwood M,
    3. Lough JRM,
    4. et al.
    (2004) Summative assessment of the competence of GP registrars 1996–2001: reasons for and implications of failure. Educ Prim Care 15:169–174, 2004.
    OpenUrl
  11. ↵
    1. Allen J,
    2. Evans A,
    3. Foulkes J,
    4. French A
    (1998) Simulated surgery in the summative assessment of general practice training: results of a trial in the Trent and Yorkshire regions. Br J Gen Pract 48:1219–1223.
    OpenUrlAbstract/FREE Full Text
  12. ↵
    1. Evans A,
    2. Nolan P,
    3. Bogle S,
    4. et al.
    (1997) Summative assessment of General Practice registrars' projects: reliability of the Yorkshire schedule. Educ Gen Pract 8:40–47.
    OpenUrl
  13. ↵
    1. Bruce L
    (1997) 97% Of GP Trainers Can't Be Wrong [letter]. Br J Gen Pract 47:529.
    OpenUrlFREE Full Text
  14. ↵
    1. Grant J
    (1996) Summative assessment for GP registrars [letter]. Br J Gen Pract 46:373.
    OpenUrlFREE Full Text
  15. ↵
    1. Campbell ML
    (1997) Letter to the Editor. Education for General Practice 8:262.
    OpenUrl
Back to top
Previous ArticleNext Article

In this issue

British Journal of General Practice: 58 (557)
British Journal of General Practice
Vol. 58, Issue 557
December 2008
  • Table of Contents
  • Index by author
Download PDF
Article Alerts
Or,
sign in or create an account with your email address
Email Article

Thank you for recommending British Journal of General Practice.

NOTE: We only request your email address so that the person to whom you are recommending the page knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Enter multiple addresses on separate lines or separate them with commas.
Summative assessment: a historical perspective
(Your Name) has forwarded a page to you from British Journal of General Practice
(Your Name) thought you would like to see this page from British Journal of General Practice.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Citation Tools
Summative assessment: a historical perspective
Stuart Murray
British Journal of General Practice 2008; 58 (557): 894-895. DOI: 10.3399/bjgp08X376357

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero

Share
Summative assessment: a historical perspective
Stuart Murray
British Journal of General Practice 2008; 58 (557): 894-895. DOI: 10.3399/bjgp08X376357
del.icio.us logo Digg logo Reddit logo Twitter logo CiteULike logo Facebook logo Google logo Mendeley logo
  • Tweet Widget
  • Facebook Like
  • Google Plus One
  • Mendeley logo Mendeley

Jump to section

  • Top
  • Article
    • BACKGROUND
    • THE MODEL: MCQ
    • CONSULTATION SKILLS
    • PRACTICAL WORK
    • TRAINER'S REPORT
    • OUTCOMES
    • LESSONS LEARNED
    • REFERENCES
  • Info
  • eLetters
  • PDF

More in this TOC Section

The Back Pages

  • How to protect general practice from child protection
  • Who Is My Patient?
  • Working with vulnerable families in deprived areas
Show more The Back Pages

Essay

  • ‘Heartsink’ patients in general practice: a defining paper, its impact, and psychodynamic potential
  • Second thoughts about the NHS reforms
  • Good enough care?
Show more Essay

Related Articles

Cited By...

Advertisement

BJGP Life

BJGP Open

 

@BJGPjournal's Likes on Twitter

 
 

British Journal of General Practice

NAVIGATE

  • Home
  • Current Issue
  • All Issues
  • Online First
  • Authors & reviewers

RCGP

  • BJGP for RCGP members
  • BJGP Open
  • RCGP eLearning
  • InnovAiT Journal
  • Jobs and careers
  • RCGP e-Portfolio

MY ACCOUNT

  • RCGP members' login
  • Subscriber login
  • Activate subscription
  • Terms and conditions

NEWS AND UPDATES

  • About BJGP
  • Alerts
  • RSS feeds
  • Facebook
  • Twitter

AUTHORS & REVIEWERS

  • Submit an article
  • Writing for BJGP: research
  • Writing for BJGP: other sections
  • BJGP editorial process & policies
  • BJGP ethical guidelines
  • Peer review for BJGP

CUSTOMER SERVICES

  • Advertising
  • Contact subscription agent
  • Copyright
  • Librarian information

CONTRIBUTE

  • BJGP Life
  • eLetters
  • Feedback

CONTACT US

BJGP Journal Office
RCGP
30 Euston Square
London NW1 2FB
Tel: +44 (0)20 3188 7679
Email: journal@rcgp.org.uk

British Journal of General Practice is an editorially-independent publication of the Royal College of General Practitioners
© 2021 British Journal of General Practice

Print ISSN: 0960-1643
Online ISSN: 1478-5242