There is increasing evidence that many determinants of quality in health care lie at the organisational rather than the individual level.^1,2 This has led to a growing emphasis on the team or organisation as the unit of analysis in quality improvement initiatives.^3–5 Policymakers increasingly seek quality initiatives that are locally owned and delivered, multiprofessional, team focused, formative, flexible, professionally led, and which draw on the known benefits of interpersonal influence and interorganisational collaboration and networking.^6–8 The Royal College of General Practitioners’ Quality Team Development (QTD) programme (box 1) is the first formal programme in England and Wales with all these key characteristics to have been systematically evaluated in a primary care setting (table 1).

The QTD programme was designed to provide a formative framework that would enable GP practices and the wider primary care team to assess the quality of the services they provide for patients and the way their team functions, and to serve as a set of benchmarking standards for use by Primary Care Organisations (PCOs—district level organisations responsible for overseeing the work of GP practices). The programme was launched in 2000 and, by October 2003, 55 PCOs (in England) and Local Health Boards (the equivalent bodies in Wales) were registered as participants.

Examples of QTD standards are given in table 2 (the full set of standards is available on the RCGP website at www.rcgp.org.uk/qtd). We sought to evaluate the design of the programme, the experience of participating practices and PCOs, and its perceived impact on patient care and teamwork.

It should be noted that, since QTD is a developmental, flexible and locally adaptable scheme, the term “evaluation” should be taken to mean “illumination of the benefits and disbenefits as perceived by participants” rather than “judgement against fixed external criteria”. Illuminative evaluation is an established approach in the social sciences that uses naturalistic (qualitative) methods to explore the rationale, development, operations, achievements, and difficulties of a particular initiative.^9–11 Parlett and Hamilton characterise illuminative evaluation as concerned with description and interpretation rather than measurement and prediction.¹² This distinction—a critical one for the evaluation of quality initiatives—is explored further in the discussion.

METHODS

The study was conducted in 2002. Four PCOs were selected (from a population of 55) to represent maximum variability in the following parameters:

• inner city v rural;

• level of deprivation;

• length of experience of QTD;

• level of financial support from the PCO to participating practices;

• level of experience of “accreditation”/quality schemes.

Semistructured interviews were conducted with 34 key stakeholders in practices and PCOs: 13 practice staff (GPs, practice nurses, district nurses, health visitors and managers) who were participating in the QTD programme but were not assessors; 14 such participants who were also QTD assessors; four PCO quality leads and managers responsible for implementing QTD; and three members of the RCGP QTD team. All interviews were conducted face to face and audiotaped with verbal consent except for four GPs who, because of time pressure, elected to give an untaped telephone interview.

The interview schedule is shown in box 2. Because this was an illuminative rather than a summative evaluation, we deliberately asked open ended questions. The questions were developed from the researchers’ own experience, a literature review on QTD and formative methods of quality improvement more generally, and discussion with others involved with quality schemes nationally. The schedule was piloted with the first PCO and minor modifications made for subsequent interviews.

Analysis of data

All taped interviews were transcribed and annotated with contemporaneous field notes. Two researchers (FM and TG) read all transcripts independently and coded responses using the constant comparative method.¹³ Each item within the data was compared with the rest of the data to establish analytical categories; negative cases that ran counter to the emerging themes were used to refine the themes. Consensus of coding categories and a final list of key themes were achieved iteratively through discussion and re-reading of transcripts and representative quotes selected to illustrate these. A preliminary version of the report was distributed to participants via the RCGP for respondent validation and the final version modified in response to feedback.

RESULTS

Response rate

The key contact in the four PCOs participating in the programme agreed to be interviewed for the study (one locality director, one locality manager, two quality leads). Of the four PCOs originally not participating in the programme, two unexpectedly signed up for it (hence making them ineligible as “controls”) and one refused to participate, citing time pressure. Only one non-participating PCO was therefore included in the study. Twelve of 14 practices approached agreed to participate and provided a total of 27 interviews (10 GPs, eight nurses and nine managers); the other two practices declined to participate, citing time pressure. There were no withdrawals among those who agreed to be interviewed. Three members of the RCGP QTD team (one GP, two administrators) were also interviewed.

The main themes of the analysis of interview data are described below.

Perceptions of QTD by respondents in GP practices

Respondents gave a variety of reasons for their practice initially signing up to QTD. Some admitted their main reason was peer pressure (but, interestingly, these respondents did not declare any subsequent regret). Others had anticipated particular benefits to their practice. Several respondents cited the breadth and flexibility of the criteria which potentially covered all areas of primary care activity which had given QTD a distinct edge over alternative schemes. Some, however, were concerned about the time commitment needed to implement the scheme.

“…all the staff changes we have had which has been impossible to set up the timing [for team building meetings]. The other problem is lack of locums. We just cannot get locums here. The last couple of sessions we ended up with one or two partners out there fending off the hoards, while the others have been up here doing the team building. It’s not really what you want, you really want everybody involved.” [GP B]

QTD was seen as a reflective developmental process that allowed their practice to develop at its own pace. They valued the fact that it engaged the whole team on real service issues and prompted breaking down of interprofessional barriers. Some respondents believed that QTD had provided a focus for coordinating practice activity and development planning. In some cases this planning process had been formalised using proprietary databases customised locally. Other respondents felt that QTD had been incidental to major changes in practice activity which they felt “would have occurred anyway”.

“I’ve been using it very much as a managerial tool and it’s been wonderful for that because I’ve been able to focus staff meetings, clinical meetings. We’ve had a point and a focus.” [Nurse A]

“We’ve sort of invested our time in matching it [QTD] effectively with the direction that we want the practice to take in its development.” [GP W]

Most respondents commented positively on the impact of QTD on team working and team relations within the practice. The programme was seen to have provided both explicit standards for the quality of teamwork and a development process that had improved teamwork. Respondents talked of improved communication and mutual understanding of roles. In some practices the programme had prompted a structured review of the contribution of different team members to existing clinics and services, leading to improved team working, more appropriate internal referrals, and better use of individual skills.

“I think it’s brought on some of the staff, because of the way it is set up and we’re trying to run it, people feel as though they are being heard more. They’ve got some input. Especially the lower staff.” [GP E]

“I think QTD will allow people to do [quality improvement] as a single portfolio rather than having loads of fragmented things that don’t kind of connect up”. [GP M]

Although respondents generally felt that their practice would have “passed” if QTD had taken a summative approach, all felt that the strengths of a formative process far outweighed the benefits of formal “accreditation”. They liked the fact that the QTD criteria were inherently developmental and hence directly relevant to all practices whatever their level of performance. A summative scheme, they felt, could induce practices to strive up to the minimum level required and not beyond. A particularly valued feature of QTD was the facility for customisation to suit local needs and circumstances. Two examples of how this occurred are shown in box 3.

Respondents felt positive about inviting colleagues in for the assessment visit and valued the feedback given. Several indicated that this was a real change in the culture of general practice, which had tended to be isolated and isolationist in the past. Nurse respondents (including those who acted as QTD assessors) were particularly positive about the value of building relationships and sharing good practice through interpractice visits which they felt had directly improved patient care and reduced the professional isolation of practice nurses.

“I have to say I was absolutely terrified, I mean it’s silly really because the girl that came I actually know her, I used to work with her. … And then she didn’t criticise at all, she was lovely. … And the assessors joined us for lunch and everybody was together and everybody [from the practice team] was talking to everybody else so that was very nice meeting.” [Nurse R]

A wide variety of changes had occurred in participating practices concurrently with the QTD programme. Some were attributed directly to participation in the programme while others “would have happened anyway”.

“I think a lot of those changes [to services] didn’t come from QTD, they were my changes… All the doctors have protected time, which hopefully improves the clinical aspect of QTD as well. But again, that wasn’t because of QTD, that was because we had invested primary care money.” [GP M]

“I think the present practice that we have works very nicely. We’ve got a nice group. We’ve always had morale I think, I don’t think it [QTD] has done anything dramatic.” [GP S]

Box 4 gives some illustrative examples, but note that this study was not designed to produce a taxonomy of such changes.

Changes in clinical services sometimes followed the self-assessment exercise but more usually followed the external visit. Respondents believed that the PCO was more responsive to bids for funding (for example, for extra practice staff) when these were tied to evidence from QTD.

Perceptions of QTD by assessors

The 15 assessors interviewed all felt that being an assessor was a positive experience and that it was a privilege to be invited into neighbouring practices. It gave them the opportunity to share good practice, pass on advice, and to learn themselves. Several respondents commented that the spread of good practice is always a two way process, with the assessment team often learning as much as the practice team. But they also felt that being an assessor involved a significant amount of work, much of it administration and paperwork.

“I would say some of the questions like, you know, ‘do you have a protocol for psychiatrically ill patients?’ well of course we don’t. Let’s have some ideas on that please and you know, if other practices have got certain things they can share it with us.” [GP S]

“I was able to share a lot with other practices. I copied all sorts of policies and documents to them that we’d already got in place and they hadn’t. I mean one practice I visited didn’t even have contracts, or job descriptions. … I felt quite hard done by that I had three fairly poor practices that I visited and the only thing I got was, I got one thing.” [Nurse A]

“Looking at our practice, when people first came round they all said what a waste of time we didn’t learn anything from it. Quite clearly the lot who came to visit us learned from it. The whole point is it’s a learning thing. Without wishing to be over modest, we are one of the best practices, so we wouldn’t expect to learn that much from it.” [GP R]

One concern raised by assessors was that it was possible to treat QTD as a paper exercise. We heard an isolated example of a two doctor partnership where the doctors “didn’t speak to each other”. One partner had allegedly completed the self-assessment form without the involvement of the practice team; staff had been coached for the assessment visit and very little had changed in the practice. There was also a more widespread concern that, once the assessment visit was over, QTD would be forgotten until the next visit and that continued support was required.

Perceptions of QTD by primary care organisations

QTD was seen by individuals in participating PCOs as a key part of their quality assurance and development programme in primary care. Like the respondents from GP practices, they valued its formative and developmental nature. They all felt pleased that all practices within their PCO had signed up to the QTD programme. This high uptake was attributed to a number of factors: local ownership, a strict confidentiality policy in which the assessment report was shared on a need-to-know basis, the RCGP “badge”, and the fact that participating practices could develop aspects of their service while not having the stigma of “failing”.

Another aspect of the programme identified by PCO respondents was the breaking down of barriers within and between practices which was felt to have reduced isolation and improved morale. One respondent cited the example of two neighbouring practices who had not communicated with one another for over a decade, supporting each other with the development of practice administrative systems. They gave numerous examples of exchange of materials and artefacts between practices (such as templates for patient letters, clinical protocols), which reduced the need to “reinvent the wheel”.

PCO respondents also valued the low cost of setting up the scheme at PCO level (around £5000 including training) and of practice visits (around £1000 per practice including indirect costs), which compared favourably with other quality improvement schemes.

A major disadvantage of QTD identified at PCO level was the heavy time commitment needed from practices, which it was felt could have implications for sustainability. Lack of assessors was also a major concern. Not all practices had GPs, nurses, or managers who were willing or suitable to be assessors, and this meant that a disproportionate workload was sometimes borne by a skeleton team of committed GPs. Finally, there were concerns about consistency and quality assurance of the assessment process. One PCO had recruited seven assessment teams, and the respondent from this PCO was concerned about variability in assessment standards, although they did not provide concrete examples.

Suggestions for improving QTD

The most common suggestion for improving QTD from GP practice respondents was to ensure the timeliness of written feedback from the assessment visit. A number of suggestions were made with a view to enhancing the validity and credibility of the assessment visit—for example, by including lay assessors on the team undertaking practice visits. Some respondents raised the issue of reassessment visits to confirm the sustainability of any improvements achieved through QTD—but they also recognised that frequent visits would be resource intensive and could be overtly disruptive.

Some practices were undergoing assessments for other schemes (such as teaching/training practice, patient access) at the same time as participating in QTD; respondents in these practices suggested aligning all such visits (and associated paperwork) to avoid duplication of effort. Finally, one respondent suggested introducing a measure of clinical competence since “QTD doesn’t actually get into the consulting room”.

The main concern of respondents at the PCO level was for quality control of the assessment visits. They felt that a formal system was needed to ensure that the assessment visits for QTD and subsequent feedback consistently conformed to a minimum standard.

DISCUSSION

This qualitative evaluation has shown that the RCGP QTD programme is generally a very positive experience for participating GP practices, who identified its main strengths as its formative, self-paced and interprofessional nature, its focus on team development, its relevance to the day to day work of primary care, its link with practice development planning, and the non-judgemental nature of the peer assessment process. This confirms and extends the findings of a small study of a pilot version of QTD.¹⁴

As shown in box 4, a wide range of service improvements have occurred in GP practices alongside QTD. Our qualitative design does not allow us to infer causality, and it is interesting that many respondents were keen to point out that changes were concurrent with—rather than caused by—QTD. We found a single example of an allegedly dysfunctional team that had failed to engage with or benefit from QTD, and one or two “flagship practices” who felt they had given more than they had gained. These findings raise (but do not prove) the hypothesis that the QTD programme acts as a catalyst but not a motivator for team development, and will provide best added value in “middle of the range” practices.

Respondents at the PCO level described how the QTD process had been an important vehicle for implementing the prevailing national policies on clinical quality and modernisation, and prompted interorganisational bridges to be built between “isolated and isolationist” GP practices. They gave examples of how ideas for improving practice had spread or co-evolved through interpractice networking.

Several GP practices in this study had introduced formal and systematic planning procedures for the first time as part of their QTD activity. The programme had helped PCOs to prioritise the use of scarce resources in funding these activities. Given the relatively primitive state of practice level strategic planning in the UK,¹⁵ this finding is potentially highly significant and should be explored further, perhaps through a quantitative survey.

Researching the team and organisational dimensions of quality improvement requires careful attention to underpinning theory,¹⁶ study design,^17–19 and the historical, social and ideological context of particular initiatives.²⁰ In a recent systematic review of the literature undertaken for the UK Department of Health we found that critical success factors for spread and sustainability of innovations in health service delivery and organisation included aspects of the innovation (its evidence base, feasibility, compatibility with existing practice and values, and potential for local customisation and reinvention); individuals’ concerns about the costs, benefits and practicalities of implementation; the nature and quality of social influence (in particular, interpersonal contact by a change agent who has homophily and credibility with those targeted for change); organisational factors (including multi-professional working, “learning organisation” culture, and absorptive capacity—that is, the organisation’s pre-existing knowledge, skills and resources); environmental factors (including political climate, prevailing organisational norms, and interorganisational collaboration and networking); and the nature and intensity of any active dissemination and implementation programmes.²¹

As table 1 shows, many of these factors are inherent to the QTD programme (which compares favourably on these criteria with other quality initiatives). However, the success of any intervention in achieving sustainable quality improvement will also depend on pre-existing organisational culture, climate, resources and priorities, and must be introduced at the right time in national and local policymaking cycles.²¹ Our study suggests that QTD “works” with the self-selecting GP practices and PCOs who volunteer for it but, of course, says nothing about how to bring non-participating organisations on board.

All quality programmes wrestle to balance the hard external values of quality control (regulation, performance management, accountability, judgement) with the soft internal values of quality improvement (development, motivation, support, ownership). GP practices in the UK can now participate in at least 10 national accreditation schemes, all of which have different ideological, theoretical, and methodological bases.²² Schemes driven by external values are often more structured and changes are easier to document against predefined criteria, but they are often widely resented, associated with target chasing and high levels of stress among participants, and do not lead to sustainable changes in practice.²³ In short, reliability is achieved at the expense of validity and acceptability. Schemes driven by internal values are more popular with practitioners and tend to produce changes that are seen as useful, but their impact and evolution is difficult to measure objectively.²³ Thus, validity and acceptability are achieved at the expense of reliability.

No scheme will fully square this circle, but QTD has many potential benefits. Our own results have highlighted some stress points in the trade off between external (summative) and internal (formative) approaches. For example, respondents valued the self-paced and non-judgemental nature of the programme, but some were rightly concerned that weaker or less motivated practices might go through the programme without ever achieving a minimum specified standard. Terms such as “assessment” and “accreditation” imply that a minimum external standard has been achieved by practices bearing the QTD kitemark—yet, in reality, the design of the programme inherently precludes hard universal standards.

Our study confirmed the finding of other researchers that quality initiatives based on peer assessment visits are hard work and that assessors, in particular, bear a heavy burden of paperwork.^24–26 The assessment process in some PCOs is being undertaken by a small number of teams, causing delays in written feedback and risking assessor burnout. However, wider recruitment of assessors without a rigorous quality control procedure could detract rather than add to the credibility of the programme. van den Homberg et al recommend that paperwork should be carefully divided into administrative data (which should be collected by non-clinical support staff) and data whose collection has a formative educational dimension (which should be collected by clinicians).²⁷ The burden to the assessment team and practice staff will thus be minimised—but only if the scheme has adequate infrastructure.

In conclusion, this preliminary evaluation suggests that the RCGP QTD programme is popular with participants and aligns well with the broader frameworks of quality improvement, quality management, and modernisation in the UK NHS^28,29 and with the specific requirements of the 2003 GP contract.⁸ We recognise the danger of extrapolating conclusions made from what is effectively a self-selecting sample. There is no research evidence to support the effectiveness of QTD in organisations that are unmotivated to embark on it, unable to resource it, and/or who lack the organisational preconditions for any planned change. Indeed, there is considerable evidence that organisations meeting this description are unlikely to make meaningful progress with any quality initiative.²¹ This study has shown that QTD is a highly complex organisational intervention that is implemented differently by different teams in different contexts at different times. As QTD is “rolled out” beyond the early adopters, further in-depth evaluation from a realist perspective—that is, designed to embrace complexity and identify “what works for whom in what circumstances”³⁰—is likely to prove particularly useful in elucidating its strengths and limitations in different contexts.