Abstract
Background The Friends and Family Test (FFT) was introduced into general practices in England in 2015 to provide staff with information on patients’ views of their experience of care.
Aim To examine the views of practice staff and patients of the FFT, how the results are used, and to recommend improvements.
Design and setting A qualitative study of a national representative sample of 42 general practices.
Method Semi-structured interviews with 43 clinicians, 48 practice managers, and 27 patient representatives. Interviews were audiotaped, transcribed, and analysed thematically.
Results Although the FFT imposed little extra work on practices, it was judged to provide little additional insight over existing methods and to have had minimal impact on improving quality. Staff lacked confidence in the accuracy of the results given the lack of a representative sample and the risk of bias. The FFT question was judged to be inappropriate as in many areas there was no alternative practice for patients to choose, patients’ individual needs would not be the same as those of their friends and relatives, and an overall assessment failed to identify any specific aspects of good- or poor-quality care. Despite being intended to support local quality improvement, there was widespread unease about the FFT, with many responders perceiving it as a tool for national bodies to monitor general practices.
Conclusion If the use of a single-item questionnaire is to continue, changes should be made to the wording. It should be focused on stimulating local quality improvement, and practice staff should be supported to use the results effectively.
INTRODUCTION
The views of patients on their experience of using health services provides information for quality improvement.1–4 In England, patients’ experiences have been measured regularly by national surveys since the late 1990s, and the findings form one of five domains of quality in the NHS Outcomes Framework.5 In primary care, the main national source of data is the annual General Practice Patient Survey.6
In May 2012, the government in England decided to introduce a Friends and Family Test (FFT) in the NHS to help patients identify the best-performing providers.7 The FFT was developed in the UK and is based on the net promoter score, which was developed in the US for use in commercial settings.8 It asks customers whether they would recommend a product or service to their friends and family. Answers are recorded on a 5-point scale from ‘extremely likely’ to ‘extremely unlikely’, and this may be followed by an open-ended question asking the reasons for that response.
In 2013, the FFT was introduced in NHS acute and maternity hospitals. In July 2014, an NHS England (NHSE) review concluded that, although the FFT had only limited value as a metric for performance management, it had the potential to promote quality improvement.9 The open-ended question was seen to be of considerable value and its inclusion became mandatory.10 Throughout 2014 and 2015, use of the FFT was expanded to the rest of health care including general practice. The question to be asked was:
‘We would like you to think about your recent experience of service. How likely are you to recommend our GP practice to friends and family if they needed similar care or treatment?’
The FFT was introduced alongside other existing methods of assessing quality, including significant event analysis, patient experience surveys, complaints, and patient participation groups (PPGs).11,12
To the authors’ knowledge, there have been no rigorous published studies of the use of FFT in primary care, hence the present study aimed at examining the views of practice staff and patient representatives of the FFT, how the results were used, and recommending improvements.
METHOD
Sampling of general practices
Forty general practices were selected from the 862 practices for which, in October 2015, reports from Care Quality Commission (CQC) inspections based on a newly introduced quality rating system for general practice were publicly available.13 Ten practices were selected from each of the four NHS regions. To maximise variation and coverage, practices were selected on size (in quartiles), CQC ratings, location (urban, rural), FFT collection method, and FFT response rate. For logistical reasons, 42 general practices were eventually recruited (Table 1).
Table 1. Characteristics of general practices selected by region
How this fits in
The Friends and Family Test (FFT) was implemented in hospitals to enable patients to choose the best-performing providers. Early review showed, however, that it was not effective for comparison across hospitals. The present study assessed the potential for use of the FFT in general practice. The FFT is responsive and easy to use, but the FFT question is inappropriately worded for general practice, results are vague, and there is misunderstanding among staff about ownership and purpose.
Interviews in general practices
Within each practice, semi-structured interviews were attempted with a clinician (GP or nurse), practice manager (or alternative), and a patient representative from the practice’s patient participation group (or local Healthwatch). In 17 practices, it was not possible to obtain an interview with a patient representative (Table 2).
Table 2. Number of practices participating by NHS region and by category of interviewee
Interviews were conducted with 43 clinicians, 48 practice managers, and 27 patient representatives (Table 3). The intention was to interview individuals separately to encourage a diversity of views, but this was achieved in only 13 practices. In 19 practices all individuals were interviewed together, and in 10 practices there were both paired and separate interviews (typically the clinician and manager were interviewed together, with the patient representative interviewed separately).
Table 3. Numbers of clinicians, managers, and patients interviewed, by NHS region
Practices were approached by letter followed by a phone call, in which the aims of the study were explained. Informed consent was sought from the participants before the interviews took place. Three interview schedules, one for each of the three roles targeted, were developed by the research team and shared with the Department of Health (DH) and NHSE. Overall, 84 participants were interviewed face-to-face and 34 by phone. Interviews were undertaken by experienced Ipsos MORI and London School of Hygiene and Tropical Medicine researchers between 5 October and 13 November 2015.
Analysis
All interviews were audiotaped and transcribed, except for four interviewees who refused to be recorded and one interview in which the recorder failed. Interviewers prepared summary notes based on the interviews in each practice, highlighting the key points that emerged.
All transcripts and interviewers’ notes were imported into NVivo (version 10). A systematic approach was employed to the analysis. This involved identification of recurrent themes by the lead researcher, which were discussed with the interviewers and research team to provide a coding framework for the full interview transcripts. Interpretation of the findings was discussed by the full team to ensure consistency and identify relationships.14 Given the qualitative nature and the sample size of the study, it was not appropriate to explore differences between subgroups of responders such as comparing the views of clinicians and patient representatives.
RESULTS
The FFT question
Most participants thought the FFT question was inappropriate for use in general practice for three reasons. First, there was concern about its phrasing given there is only one general practice accessible in some parts of the country, so there is no choice. Asking a patient to recommend a particular practice appears out of place and is potentially confusing. Moreover, given that patients may have no experience of other practices, it may be difficult for them to make a comparative judgement about their own practice.
Second, given the personal nature of healthcare needs, it is unlikely that friends or family members will have the same needs. Also, the relationship between a patient and practice staff usually plays a role in determining levels of satisfaction, but a patient cannot assume that friends and family will experience a similar relationship.
Third, there was concern about the lack of detail in the answers. A practice performs a wide range of activities to respond to the specific needs of each patient, so the anonymised and generic feedback provided by the FFT is of limited value in identifying what activity the patient may have found unsatisfactory.
‘I can understand if you are in a city, and you’ve got choices … But if you are in a village or in a very rural area it’s a completely pointless exercise.’
(Patient representative, practice 19, rural)
‘Well I’m not sure recommending the practice is the most important issue to patients, is it, whether they’ll recommend it to someone else? The most important issue to them is whether or not they’ve got a good GP and they feel like they’re going to be looked after properly.’
(Patient representative, practice 13, urban)
‘Unless they come and tell you their name, I can’t follow it up. I can’t make it better because it’s not specific enough for me to be able to think, right, OK, on that day this is what happened.’
(Practice manager, practice 17, urban)
Understanding the aim of the FFT
Staff were often unclear about the reasons for implementing the FFT. Many assumed that it was collected by national bodies (for example, NHSE, DH) to monitor the quality of care provided, and, possibly, to take action where results were poor. The mandatory requirement to provide monthly returns was perceived as evidence of this. Even the FFT forms and collection box could be perceived as ‘belonging’ to the government, being placed in the practice by the NHSE to pursue its own ends.
This widespread belief was associated with staff generally perceiving the FFT as something they were required to do on behalf of the government rather than in the interests of the practice. For many practices, the only reason for implementing the FFT was to comply with contractual requirements:
‘Because it’s mandatory … Because we’ve been told its contractual, and it has to be reported though CQRS every month … We’re given the dates on which the data has to be in. Why do they want the data? Well I guess it’s a measurement of how good, bad, or indifferent the practice is from the central point of view.’
(Practice manager, practice 17, urban)
This perception generated unease, given the doubts of staff about the validity of the data collected. There was a consensus that the low number of responses at practice level meant that FFT results were unlikely to be representative of practice patients or to provide reliable indicators of service quality. There was awareness that the patients who completed the FFT were self-selected or, maybe, handpicked by staff (where paper forms were used).
There was concern that a few critical comments might provide a distorted picture, whereas others pointed out that positive feedback could be influenced by the asymmetry of information or the power imbalance between the practice and its patients.
FFT and the gatekeeping role
The perception of the FFT as a centralised monitoring tool, combined with the very low response rate casting doubt on the generalisability of the results, contributed to the view that its use was susceptible to patients who wished to ‘punish’ practices by giving a low FFT rating if they did not get what they felt they needed or wanted. This perception highlights a potential conflict with the practice’s gatekeeping role and perhaps resonates with a wider range of policies in which practices are increasingly requested to act as patients’ agents in a quasi-customer/retailer relationship:
‘It is a measurement of client, patient happiness as to [whether] the consultation’s gone the way that they wish it to … So if I could get someone to give me a two from refusing an inappropriate antibiotic, well, that’s clever of me. But, yeah, it’s not a measure of the efficiency of service.’
(GP, practice 6, rural)
Some even felt that the implementation of the FFT responded to a political decision to punish GPs:
‘They’re trying to show the public that the government is going to beat primary care with a big stick.’
(Practice manager, practice 20, urban)
The FFT and other feedback collection tools
Many interviewees felt that the FFT provided little information of value, especially for practices that had other ways of collecting patient feedback. A large number mentioned their practice’s own patient survey as being more effective in identifying shortcomings in quality. Other types of feedback included formal complaints and the quality of their personal relationships with patients.
Moreover, the open-ended comments were reported to be quite generic and lacking detail, which reduced their value in identifying and addressing quality issues:
‘It’s not telling us anything we don’t already know. If the practice can’t make use of it, I don’t see the point in collecting it. Because we do surveys twice a year, a more detailed survey where you’re asking specific questions. ’
(Practice manager, practice 30, urban)
‘I think the other surveys we do are probably a little bit more detailed so you get down to more specific information if there is a problem … The [FFT] comments are quite generic, so there’s no real … you don’t get the detail of information that probably would influence you to make changes, as of yet. ’
(Practice manager, practice 19, rural)
Impact of the FFT
In only four of the 42 practices were positive views about the FFT expressed and in only one was an example of how the results had led to improved quality mentioned:
‘There’s nothing wrong with that little questionnaire other than it’s useless.’
(GP, practice 22, urban)
‘There was one comment we had about somebody with difficulty getting a wheelchair from the car park, so we used that to ensure the landlord changed the way the ramp is in the car park, so it gave us a bit of ammunition and it worked as a leverage to allow change for the better.’
(Practice manager, practice 28, urban)
Quality improvement in general practice
On a more general note, significant variability was observed in the extent to which general practices are committed to using quality assessments for quality improvement. A few practices were well advanced, having set up effective PPGs and appeared to make good use of local surveys. Other practices, however, seemed to struggle in this respect, partly reflecting a resistance to change, and limited resources and knowledge of how to respond to assessments showing less than optimal quality.
DISCUSSION
Summary
Although the FFT imposed little extra work on practices, it was judged to provide little additional useful insight over existing methods and to have had little or no impact on helping to improve the quality of services. Staff lacked confidence in the accuracy of the results given the low response rate and unrepresentative nature of responders.
The question used in the FFT was judged to be inappropriate as in many areas there was no alternative practice for patients to choose, a patient’s individual healthcare needs would not be the same as those of their friends and relatives, and asking for an overall assessment failed to identify any specific aspects of good- or poor-quality care. Despite being intended to support local quality improvement, there was widespread unease about the FFT, with many responders perceiving it as a tool for national bodies to monitor (and criticise) general practices.
Strengths and limitations
The strengths of this study included the diversity of the sample in terms of practice size, location (region and urban/rural), assessed quality (CQC rating), and FFT collection methods. Data were also collected from all the main groups involved, namely, GPs, practice administrative staff, and patients. The study also took place 10 months after initiation of the FFT in general practice, which allowed enough time for practices to become familiar with its implementation and be in a position to judge how it was functioning and its likely contribution to quality improvement.
There were four limitations. First, on average, the participating practices collected a larger number of FFT responses than all practices in England, suggesting that they were more engaged with FFT than those that did not participate. So it is possible that responders were more positive than might be found throughout primary care. Given the generally negative tone detected, the present results might overestimate the support for FFT. Second, the views of patient representatives were those of people who were involved to some degree in the running of a practice. Their views may not, therefore, be typical. The failure to interview a patient representative in some practices may reflect staff achieving less patient engagement. Such practices might be less concerned about the views of their patients, which might mean the views obtained did overestimate support for FFT.
Third, the perceptions of staff were sought and reported rather than observations made of what actually takes place, which might be different. Fourth, as interviews in practices included both clinical and non-clinical staff, and occasionally patient representatives as well, this may explain why no significant differences in views between different types of interviewees were found.
Comparison with existing literature
Despite significant differences in implementation, namely the absence of targets and financial incentives associated with achieving higher response rates, many of the concerns and views about the FFT expressed by staff and patients in general practice are consistent with those previously observed in acute hospitals. Past hospital research showed that the FFT is vulnerable to selection bias, making the quantitative data unfit for comparisons across providers.9,15
It was also found that managers from acute and community trusts believed that excessive emphasis was put on the central assurance process rather than on enabling local analysis of qualitative data and quality improvement.16
Implications for research and practice
It is believed that the principal policy challenge is whether to persist with the FFT (or a similar single-item questionnaire) or not. If such an approach is favoured, this study suggests there are four ways to enhance its value. First, the content of the FFT could be changed. A simpler and more straightforward question that does not include a reference to ‘recommendation to friends and family’ would probably provide a better measure of patients’ experiences. In addition, the data generated could be of greater use for quality improvement if practices were encouraged to collect patients’ views on specific aspects of services. Obtaining feedback on topics of concern for a practice could work as a quick diagnostic tool to make staff aware that a problem exists when negative and consistent feedback is received, and would provide more detailed and timely information on existing quality issues, possibly filling the gaps that may be left uncovered by other approaches.
Second, there is a need to improve practice staff understanding of the purpose of the FFT. The mechanism of monthly data returns seems to be one of the main factors leading to confusion about the purpose of the FFT. This has also hindered the perception of the FFT as a tool that belongs to the general practices and that can help them improve their services. Considering the limited usefulness of the quantitative data provided by the FFT, the DH and NHSE may reduce or eliminate monthly reporting to encourage local ‘ownership’ and use of the FFT. The more demands are made by the centre, the less the feelings of local ‘ownership’. Removing mandatory monthly reporting would dispel the idea that the FFT was meant to be used by ‘difficult’ patients against practices, which was disheartening for many, and would avoid any contradiction between practices’ gatekeeping role and their desire to satisfy their patients. Asking for reports on the quality improvement activities carried out by general practices might be a suitable alternative.
Finally, there is a need to increase the capacity of general practices to manage quality, which goes beyond the use of FFT. Support and guidance on how to set up local surveys and PPGs, and how to maximise their contribution to quality improvement initiatives, should be part of any strategy. Suggestions about how to improve quality were included in the FFT implementation guidance for NHS-funded services, but not in that provided for general practices, which may benefit from specific guidance on this aspect.10,17 A body of literature exists on effective methods and techniques that can be used in general practices to improve quality,18 and more could be generated from further research.
Acknowledgments
The authors thank the members of staff and patient group representatives in all the general practices who kindly agreed to give up their time to be interviewed for this project. Thanks also to the research team at Ipsos MORI — Debbie Lee Chan, Rory Donaldson, Harriet Fowler, Josie Lloyd, and Ewa Ochmann — for their significant contribution to developing and testing the interview topic guides, carrying out the fieldwork with general practices, preparing summaries of the interviews, and for their observations and discussion of the emerging findings.
Notes
Funding
This study is part of the programme of the Policy Innovation Research Unit (http://www.piru.ac.uk/). This is an independent research unit based at the London School of Hygiene and Tropical Medicine, funded by the Department of Health Policy Research Programme (contract 102/1001). Sole responsibility for this research lies with the authors and the views expressed are not necessarily those of the Department of Health. The Department of Health played no role in the design of the study, the interpretation of the findings, the writing of the paper, or the decision to submit.
Ethical approval
Granted by the Research Ethics Committee of the London School of Hygiene and Tropical Medicine (REC reference 10283). As the study was eligible for Cohort 1 of the stepped implementation of Health Research Authority Approval process, permission for all NHS sites involved in the study was granted through a single application. The application was made through the IRAS online form (IRAS Project ID: 186617) and approved by HRA on 26 August 2015.
Provenance
Freely submitted; externally peer reviewed.
Competing interests
The authors have declared no competing interests.
Discuss this article
Contribute and read comments about this article: bjgp.org/letters
