Abstract
Background National standards for delivery of out-of-hours services have been refined. Health service users' preferences, reports, and evaluations of care are of importance in a service that aims to be responsive to their needs.
Aim To investigate NHS service users' reports and evaluations of out-of-hours care in the light of UK national service quality requirements.
Design Cross sectional survey.
Setting Three areas (Devon, Cornwall, Sheffield) of England, UK.
Method Participants were 1249 recent users of UK out-of-hours medical services. Main outcome measures were: users' reports and evaluations of out-of-hours services in respect of the time waiting for their telephone call to the service to be answered; the length of time from the end of the initial call to the start of definitive clinical assessment (‘call back time’); the time waiting for a home visit; and the waiting time at a treatment centre.
Results UK national quality requirements were reported as being met by two-thirds of responders. Even when responders reported that they had received the most rapid response option for home visiting (waiting time of ‘up to an hour’), only one-third of users reported this as ‘excellent’. Adverse evaluations of care were consistently related to delays encountered in receiving care and (for two out of four measures) sex of patient. For 50% of users to evaluate their care as ‘excellent’, this would require calls to be answered within 30 seconds, call-back within 20 minutes, time spent waiting for home visits of significantly less than 1 hour, and treatment centre waiting times of less than 20 minutes.
Conclusion Users have high expectations of UK out-of-hours healthcare services. Service provision that meets nationally designated targets is currently judged as being of ‘good’ quality by service users. Attaining ‘excellent’ levels of service provision would prove challenging, and potentially costly. Delivering services that result in high levels of user satisfaction with care needs to take account of users' expectations as well as their experience of care.
- out-of-hours medical care
- primary care
- quality of health care
- questionnaire
- satisfaction
- standards
- unscheduled care
INTRODUCTION
The NHS has defined criteria and standards of attainment for providers of out-of-hours services.1 These standards were developed following a national review of unscheduled care provision2 and defined after consultation with the Royal College of General Practitioners, while taking account of government policy.3 The development of the out-of-hours patient questionnaire, a valid and reliable survey instrument suitable for evaluating patients' experience of out-of-hours services within the UK, has recently been described.4
Patient's experiences of, and satisfaction with care are increasingly used to inform models of service delivery.5 From a theoretical perspective, it has been suggested that users' views of health services can be described as either preferences, evaluations, or reports of their experiences.6 Preferences assess patients' choices and desired outcomes regarding their health care, and, as such, can be assessed in samples of responders who may or may not have experienced the services under consideration. In contrast, both evaluations and reports are based on direct experiences of care. Evaluations seek patients' views on their experiences of healthcare services and of the adequacy of the care provided, while reports refer to observations regarding the organisation or process of care, which are divorced from statements regarding the adequacy of the care provided. In this study, users' reports of their experiences were examined and related to users' evaluations of the care they received, focusing on four nationally designated standards of out-of-hours care.
METHOD
As part of the development process of the out-of-hours patient questionnaire, three English out-of-hours service providers invited consecutive service users to complete a postal questionnaire about their experience of the service.4 Samples were drawn from three volunteer sites in south-west and northern England (Cornwall, Devon, Sheffield). These areas were identified as they work across a range of urban, rural, and inner-city patient populations with diverse community profiles in respect of geography, ethnicity, and demography. A different out-of-hours care provider organisation provided care at each of the three sites.
All three adopted models of care, which were similar, have been previously described in detail.4,7 The size of the sample was sufficient to support the process of questionnaire design.8 The study took into account best practice and guidance in the conduct of postal surveys of health users9,10 within resource constraints. Consequently, personalised letters to potential participants, in coloured ink, were sent by first class post with a stamped-addressed return envelope.
Following ethical approval in September 2005, sampling took place within 2 weeks of out-of-hours contacts. To minimise bias users requesting care at weekends and/or evenings were included in each area. Detailed methodology of the survey process, including the rationale for excluding around 30% of calls (for example, nursing-home residents and those with end-stage terminal illness) from the sampling frame, is described elsewhere.4 Contact details of a consecutive sample of 1249 recent calls to the out-of-hours service providers were extracted, and providers sent a numbered questionnaire, information sheet, and reply-paid envelope to the patient (or parent or guardian if the patient was a child). A reminder questionnaire was sent to non-responders after 2 weeks. Participants reported their experience of care in terms of making contact with out-of-hours services; the outcome of their contact; the consultation with the health professional; and their experience of home visiting or treatment centre attendance (if relevant). Within each domain, responders were asked to report on their experience of care, and then (separately) to evaluate their experience using a 5-point scale.
How this fits in
The NHS has defined criteria and standards of attainment for providers of out-of-hours services. Users' reports of their experiences of out-of-hours care were examined and related to their evaluations of the care they received. UK out-of-hours service users have exacting standards in respect of their care which may prove challenging for providers to meet.
Participant responses were examined in the light of four of the 13 nationally-designated requirements defined for service delivery by the NHS.1 The four standards (Box 1) related to the time taken in answering the telephone call to the service (standard 8); length of time to the start of definitive clinical assessment (‘call back time’, standard 9); reported waiting time before receiving treatment when attending a treatment centre (standard 10); and time taken in waiting for a home visit (standard 12). These standards were selected as they could be reported and evaluated by patients based on their own, direct experiences; the other requirements focus more around the processes by which providers must operate (for example, reporting arrangements between providers and commissioning trusts).
The proportion of patients reporting their experiences of out-of-hours care that mapped onto the response target appropriate for that standard are presented. Relationships between reports of care and evaluations of that care were also examined to determine the levels of care required to meet a certain level of patient satisfaction; for example, how quickly do calls have to be answered to ensure that 50% of patients evaluate the service as ‘excellent’.
The contribution of responders' age (0–4, 5–11, 16–64, ≥65 years), sex (male, female), and socioeconomic status (using the Townsend score quintile derived for each participant's postcode) in predicting their evaluations of service provision was investigated for each of the four national requirement items. A further variable was added accounting for the location (three centres) from which the user received care. To develop the model, users' evaluations in respect of the four items of care were dichotomised (‘very poor, poor, or acceptable’ versus ‘good or excellent’). This categorisation was selected as the cut-point and was broadly similar to that reported elsewhere.11 Binary variables categorising reports regarding whether or not the national quality requirements had been met were also analysed. Variables that were significantly associated on univariate analysis with users' evaluations of care (P<0.05) were entered into a binary logistic regression model as potential explanatory variables. Where appropriate, adjusted odds ratios were calculated for variables independently predicting users' evaluations of the service. All analyses were carried out using SPSS (version 15.0).
RESULTS
Median patient age of the survey population (n = 1249) was 40 years (interquartile range = 19–64 years; Table 1). Approximately two-thirds (745/1249; 59.6%) were female. Area three differed from the other two sites with the mean patient age approximately 10 years younger, and higher deprivation scores (indicating greater deprivation) when compared with the other two areas sampled. Five hundred and sixty-four (45.2%) patients were treated at a treatment centre, 441 (35.3%) received telephone advice from a health professional, 206 (16.5%) had home visits, and 38 patients (3.0%) had their problem dealt with by the call operator.
Following one reminder to non-responders, 627/1249 individuals returned questionnaires (50.2% response rate), of whom 570 (45.6%) provided useable data. Patients providing useable data (Table 2) were older and more affluent than non-responders, although both groups had a similar sex profile. Of 464 individuals who provided information on ethnicity, 452 (97.4%) reported their ethnic status as white.
Users' reports of service provision suggested that national quality requirements were attained according to around two-thirds of service users for three of the four requirements investigated (Table 3). For the remaining requirement (time spent waiting for an answer to the initial call; standard 8), nearly 87% of users reported that the national quality standard was attained. While more than 80% of users who reported that their care had been delivered within the national quality requirement evaluated their care as being of at least a ‘good’ standard across three of the four criteria examined (and 72.2% for the other requirement), much smaller proportions of these users evaluated their care as being ‘excellent’. Based on these observations, for 50% of users to evaluate their care as ‘excellent’ this would require calls to be answered within 30 seconds, call-back within 20 minutes, time waiting for home visits of significantly less than 1 hour, and treatment centre waiting times of less than 20 minutes.
Binary logistic regression models are presented in Table 4. Univariate analysis identified that age, socioeconomic status, and location of care were not associated with participants' evaluations of care. Female sex was associated with adverse evaluations of both the time experienced in waiting for a call-back from a health professional (168/266 females [63.2%] providing evaluations of ‘good’ or ‘excellent’ compared with 131/182 males [72.0%]), and of the time experienced in waiting for a home visit to take place (27/74 females [36.5%] providing evaluations of ‘good’ or ‘excellent’ compared with 47/71 males [66.2%]), but not in respect of the length of time encountered in getting through to the service or in the length of time encountered when waiting to be seen following attendance at a treatment centre. For all four standard-related questionnaire items, univariate analysis identified that reports of longer time waiting were associated with adverse evaluations of care. All of the candidate variables were identified as predictors of users' evaluations of the care which they had received.
DISCUSSION
Four items relating to national requirements for out-of-hours service provision were incorporated in this questionnaire. On the basis of the results, services are meeting current standards in their management of the majority of patients. However, achieving ‘excellent’ levels of user satisfaction (the level to which services and the Department of Health might aspire) would prove extremely exacting for service providers.
Delays encountered in receiving out-of-hours care were consistently identified as independent predictors of adverse evaluations of care. Even if telephone calls were reported as being answered within 30 seconds, only around two-thirds of users reported this as an ‘excellent’ level of service provision. Where the service response was between 30 and 60 seconds, only around one-fifth of users reported this provision as being ‘excellent’. Where a less stringent target of attainment was set (service provision being ‘good’ or ‘excellent’), much higher levels of satisfaction with the experience of out-of-hours care were evident.
Female sex was identified as an independent predictor of adverse evaluations in respect of delays encountered in two of the four quality requirement domains investigated: the time spent waiting for a health professional to call back and the time spent waiting for a home visit. McKinley and Roberts12 have previously noted that female sex was associated with adverse evaluations of out-of-hours care. The potentially complex contribution of sex as a determinant of satisfaction with out-of-hours care (such as the relationship between sex and carer status) is an area that would benefit from further carefully planned and research modelling.
An earlier report derived from this study revealed that age was a predictor of satisfaction with the out-of-hours consultation, but was not a predictor of satisfaction with those arrangements established to provide access to the service.4 In the current study age was not an independent predictor of adverse evaluations in terms of delays encountered in accessing the out-of-hours service. While many studies have identified that older individuals tend to express greater satisfaction with health care than younger individuals, the current results suggest that they are equally reluctant to accept delays in service provision.
Strengths and limitations of the study
This study was based on a large sample size, using participants drawn from three geographically distinct areas of the UK. Following only one reminder, although the survey response rates were somewhat low, responses rates were in line with those reported by recent community-based surveys from the UK13–15 and elsewhere.16,17 While low response rates can introduce bias,6,18 it is not possible to predict the magnitude or direction of the effect with any certainty, as the limited previous research is contradictory. For example, there is some evidence that response bias suggests an under-reporting by individuals who had less satisfactory experiences,6 although this is counter-balanced by findings that postal surveys (such as adopted in this study) tend to give more variable responses, as people who feel strongly either way are more likely to respond. Despite sampling in an area of high ethnic minority representation, individuals from ethnic minorities were underrepresented in the responder sample compared to UK normative data,19 and older, more affluent responders were overrepresented. The sampling also excluded some service users on ethical grounds, such as individuals with end-stage terminal illness or children aged 12 to 16 years, limiting the generalisability to these groups.
The research is based on patients' reports of out-of-hours service provision, combined with subsequent evaluations of the adequacy of the recollected experiences. The validity of these reports was not investigated in detail by, for example, cross checking patient reported waiting times with process data recorded in healthcare provider computer systems. Other studies from different service settings have raised questions regarding the validity of patient reports of healthcare utilisation,20 noting a tendency for inaccuracy of recall21 and under-reporting of service use, especially among older service users22,23 and over longer periods of recall-time such as 6 or 12 months.22,24 The current research was based on questionnaire surveys completed within 2 weeks4 of a contact with the out-of-hours services, and thus may be subject to less recall-bias than studies which are dependant on longer periods of recall.
National quality requirements1 for the delivery of out-of-hours services set explicit standards for service provision across 13 domains. Patients' reports and evaluations of care were examined for only four of these domains, selecting only those areas which could feasibly be included within a questionnaire relating to the direct experiences of service users. Furthermore, the national requirements make clear that service provision may be matched to patients' needs in those services operating a system for prioritising patient contacts following an initial assessment of their clinical need. Since the aim of the study was to gain an overview of patients' experiences of service delivery rather than to focus on individual patient care, the classification of the urgency of calls undertaken by call handlers in their preliminary assessments of patients' needs was not taken into account.
This study demonstrates the value of combining patients' reports of care with questions inviting their evaluation of out-of-hours care. This process offers potential in informing standard setting and service configuration.24,25
It is clear that UK out-of-hours service users have high expectations in respect of their quality of care. While users' reports and evaluations are predicated on their experience of care, patient preferences for care have been proposed to be independent of experience.6 Although patient preferences have been considered in the arena of clinical research,26,27 it is suggested that patients' expectations represent a distinct further area for consideration in a model of patients' views of health services. Furthermore, it is speculated that satisfying user preferences for care may be more difficult than meeting their expectations of care, as the latter is moderated by both personal experience and the wider social and political context in which care is delivered. Recognition of the role of user expectations as a mediator of patient reports of satisfaction has been previously reported.28 While there is reasonable consensus that such expectations play an important role in understanding users' judgements of service quality and satisfaction,18 there remains considerable theoretical uncertainty within psychology as to how cognitive and affective processes combine when making such evaluations.
In this study, service users whose care did not meet publicised targets of service provision reflected this in adverse satisfaction ratings. A previous study of out-of-hours care29 has reported that meeting patients' expectations of care is a major independent predictor of patients' satisfaction with care. Satisfaction with care may reflect the size of the gap between patients' expectations and their experience of care with a smaller gap being associated with improved satisfaction. This simple but important hypothesis requires testing with prospectively collected data in studies designed to address the research question. If correct, the hypothesis suggests that improved reports of satisfaction with care may be achieved by improving patients' experience of care, but also by setting and communicating realistic targets in respect of healthcare delivery with a view to informing patients' expectations of health service delivery.
Acknowledgments
We are grateful for the support of out-of-hours service providers and service users who contributed to this study.
Notes
Funding body
UK Department of Health
Ethical approval
North and East Devon NHS Research Ethics Committee (126/12/2004)
Competing interests
Michael Greco is a director of Client Focused Evaluation Programme (UK). All other authors have stated that there are none.
Discuss this article
Contribute and read comments about this article on the Discussion Forum: http://www.rcgp.org.uk/bjgp-discuss
- Received September 12, 2008.
- Revision received November 7, 2008.
- Accepted November 27, 2008.
- © British Journal of General Practice, 2009.