Skip to main content

Main menu

  • HOME
  • ONLINE FIRST
  • CURRENT ISSUE
  • ALL ISSUES
  • AUTHORS & REVIEWERS
  • SUBSCRIBE
  • BJGP LIFE
  • MORE
    • About BJGP
    • Conference
    • Advertising
    • eLetters
    • Alerts
    • Video
    • Audio
    • Librarian information
    • Resilience
    • COVID-19 Clinical Solutions
  • RCGP
    • BJGP for RCGP members
    • BJGP Open
    • RCGP eLearning
    • InnovAIT Journal
    • Jobs and careers

User menu

  • Subscriptions
  • Alerts
  • Log in

Search

  • Advanced search
British Journal of General Practice
Intended for Healthcare Professionals
  • RCGP
    • BJGP for RCGP members
    • BJGP Open
    • RCGP eLearning
    • InnovAIT Journal
    • Jobs and careers
  • Subscriptions
  • Alerts
  • Log in
  • Follow bjgp on Twitter
  • Visit bjgp on Facebook
  • Blog
  • Listen to BJGP podcast
  • Subscribe BJGP on YouTube
Intended for Healthcare Professionals
British Journal of General Practice

Advanced Search

  • HOME
  • ONLINE FIRST
  • CURRENT ISSUE
  • ALL ISSUES
  • AUTHORS & REVIEWERS
  • SUBSCRIBE
  • BJGP LIFE
  • MORE
    • About BJGP
    • Conference
    • Advertising
    • eLetters
    • Alerts
    • Video
    • Audio
    • Librarian information
    • Resilience
    • COVID-19 Clinical Solutions
Research

Referral of patients with chronic obstructive pulmonary disease to pulmonary rehabilitation: a qualitative study of barriers and enablers for primary healthcare practitioners

Jane S Watson, Peymane Adab, Rachel E Jordan, Alexandra Enocson and Sheila Greenfield
British Journal of General Practice 2020; 70 (693): e274-e284. DOI: https://doi.org/10.3399/bjgp20X708101
Jane S Watson
Faculty of Health, Medicine, Education & Social Care, Anglia Ruskin University, Cambridge; Institute of Applied Health, University of Birmingham.
Roles: Senior lecturer in community nursing
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Peymane Adab
Institute of Applied Health, University of Birmingham.
Roles: Professor of chronic disease epidemiology and public health
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Rachel E Jordan
Institute of Applied Health, University of Birmingham.
Roles: Reader in epidemiology
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Alexandra Enocson
Institute of Applied Health, University of Birmingham.
Roles: Senior programme manager
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Sheila Greenfield
Institute of Applied Health, University of Birmingham.
Roles: Professor of medical sociology
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Article
  • Figures & Data
  • Info
  • eLetters
  • PDF
Loading

Abstract

Background Pulmonary rehabilitation (PR) is a cost-effective, internationally recommended intervention for patients with chronic obstructive pulmonary disease (COPD). Referral is predominately led by primary healthcare practitioners (PHCPs), but referral and patient uptake is poor.

Aim To understand barriers and enablers for PHCPs when considering patient referral to PR, to explore the influence of patient characteristics, and to understand how referral rates may be increased.

Design and setting PHCPs who care for and refer patients with COPD to PR were purposively selected from general practices across Cambridgeshire and Peterborough, and the West Midlands.

Method A qualitative study. Semi-structured interviews were undertaken to theme saturation, exploring PR referral. Images depicting patients with varying COPD severity were used to stimulate memory and associative recall. Interviews were recorded, transcribed verbatim, and analysed using rapid qualitative analysis.

Results A total of 19 PHCPs were interviewed. Barriers to PR referral included limited awareness of the clinical benefits, little knowledge of local PR providers, consultation time constraints, and presumed low patient motivation. While practice nurses had the greatest knowledge, they still described difficulty in promoting PR. PHCPs frequently described assessing patient suitability based on presumed accessibility, social, and disease-specific characteristics rather than the clinical benefits of PR. Referrals were facilitated by financial incentives for the practice and positive feedback from patients and providers.

Conclusion There were more barriers to PR referral than enablers. Providers must engage better with PHCPs, patients with COPD, and carers, and actively promote PR. Increasing PHCPs’ awareness of the benefits of PR, financial incentives, and alternative referral pathways should be considered.

  • COPD
  • pulmonary disease, chronic obstructive
  • pulmonary rehabilitation
  • referral

INTRODUCTION

Chronic obstructive pulmonary disease (COPD) is a chronic debilitating disease, the fifth leading cause of death in the UK.1 Emergency hospital admissions remain high,2 and annual general practice COPD consultations have risen from a mean of 6.6 per patient with COPD in 2002 to 12.7 in 2010.3

Pulmonary rehabilitation (PR) is a cost-effective, nationally, and internationally recommended intervention for patients with COPD.1,4–6 It comprises structured multidisciplinary group sessions that combine individualised exercise with disease-related education.7 Studies consistently demonstrate that PR improves patients’ symptoms, in stable disease and post-exacerbation.8–11 Box 1 summarises the main benefits of PR experienced by patients with COPD.

  • Health-related quality of life improved (Chronic Respiratory Disease Questionnaire & St George’s Respiratory Questionnaire)

  • Dyspnoea decreased

  • Fatigue decreased

  • Emotional function increased

  • Mastery increased

  • Symptoms reduced

  • Disease impact reduced

  • Exercise capacity increased

  • Hospital readmission likely to decrease (moderate evidence)

  • COPD = chronic obstructive pulmonary disease.

Box 1.

Benefits of pulmonary rehabilitation in stable disease and post-COPD exacerbation requiring hospital admission10,11

In the UK, referral is largely undertaken by primary healthcare practitioners (PHCPs),12 but less than 10%–15% of eligible patients with COPD are being referred, consequently practitioners are being asked to improve PR promotion.12–14

Qualitative studies among patients with COPD repeatedly highlight barriers that impede referral acceptance, include fears around exercise and breathlessness, and feelings of guilt. Functional barriers include accessibility of PR, and impacts on family and work.15–21 Patients also report their referral acceptance is highly influenced by their HCP, citing information about PR, referrer’s manner, and professional conviction about PR as important.15–18,21–23

To date, only two UK primary care studies13,24 have sought to understand PHCPs’ PR referral experiences. These studies suggest that time constraints, limited knowledge of PR, patient’s physicality, and personal culpability for COPD inhibit referral acceptance. Both studies suggest PHCPs’ attitudes to PR could be important reasons for low referral numbers.

However, one of the studies,24 based on five focus groups with 21 HCPs from three general practices in a limited geographical area was conducted over a decade ago when PR availability was limited.25 The second13 was based on a pre-post evaluation of multiple interventions to overcome low PR referral, using semi-structured surveys with HCPs. While some potentially useful strategies were identified, the small number of participants (nine out of 22 responded to follow-up), shortcomings in the survey design, and data analysis process limit interpretation of the study findings.

A 2018 systematic scoping review, which included a summary of articles exploring HCPs’ perceived barriers and enablers to PR referral,26 identified nine worldwide studies including two from the UK.13,24 Poor knowledge of PR was a commonly perceived barrier, with a need for education identified as a potential enabler. However, the included studies were not limited to primary care settings (a setting which is particularly varied worldwide); included surveys that limit exploration of the full range of potential barriers; and were dominated by several studies from the same researchers.27–29 The design and contextual variations limit interpretation of PHCPs’ PR referral experiences relevant to a UK primary care setting. Such understanding is critical for informing interventions to increase referral rates.

Referral to pulmonary rehabilitation (PR) is an important activity, yet it is infrequently carried out by primary healthcare staff. It is suggested that increasing healthcare professionals’ knowledge and awareness of PR will positively change their referral behaviour; however, this qualitative study highlights many complex intertwined barriers to PR referral and few enablers for primary healthcare staff in England. The findings highlight some of the factors that should be addressed in addition to interventions targeting knowledge and raising awareness, to increase PR referral rates.

How this fits in

This study of PHCPs’ perceptions of barriers and enablers for referral of patients with COPD to PR and the influence of patient characteristics on behaviour aims to provide an up-to-date exploration of the referral process, which will inform the development of potential interventions to improve primary care PR referral.

METHOD

The study was undertaken in general practices in Cambridgeshire and Peterborough, and the West Midlands (NHS Birmingham South Central clinical commissioning group [CCG] and NHS Birmingham CrossCity CCG). It was not the intention to identify differences, but to describe the practices and experiences in two contrasting regions.

The prevalence of COPD across the localities is broadly similar, with rates for 2015–2016 of 1.45%, 1.65%, and 1.67% in NHS Birmingham South Central CCG, NHS Birmingham CrossCity CCG, and NHS Cambridgeshire and Peterborough CCG, respectively.30 However, spend per 100 000 on COPD emergency admissions in 2014–2015 was higher in the West Midlands.31 Additionally, Birmingham is one of the most ethnically diverse and deprived regions in the UK,32,33 which is associated with increased primary care consultations.34

There were nine PR programmes available in Cambridgeshire and Peterborough, and seven in the West Midlands, but given the West Midlands’ larger population,35 PR provision per head is less in this location.

Sampling

Fifty-three practice managers from a total of 272 practices within the two regions were randomly selected and PHCPs interested in respiratory and/or COPD within these practices (n = 136) were identified through practice websites and invited by email to participate. Invitations contained participant information sheets and reply slips that collected data on job role, estimated number of COPD patients seen weekly, and referrals to PR. A follow-up email was sent to non-responders.

Adaptive sampling methods were implemented to enhance initial recruitment.36 This included one author attending three regional respiratory-focused meetings across Birmingham and Cambridgeshire to promote the study verbally. Link-trace sampling approaches were also used,37 with invited participants suggesting further potential invitees.

Purposive sampling was undertaken to ensure a fair representation of job role in each of the two locations.38 Practices were remunerated for participant time when interviews took place within working hours.

Data collection methods

Semi-structured face-to-face39 or telephone interviews40 were offered to PHCPs. All interviews were conducted by one author, audiorecorded, transcribed verbatim, checked, and anonymised. Validation of completed individual transcripts was requested from participants via email to ascertain content accuracy and enhance study reliability.41

The topic guide (Box 2) was informed by published literature and encompassed questions around capability, opportunity, and motivation to understand influences on HCP behaviour.42 This guide was piloted and revised by two authors before study commencement.

  1. Could you tell me in what context do you currently see COPD patients? (for example, planned — annual review/flu jab or unplanned — exacerbation)

  2. On average how many COPD patients do you think you see per week?

  3. Do you currently refer to pulmonary rehabilitation programmes?

  4. What is your understanding/view surrounding pulmonary rehabilitation programmes in general?

    … And in relation to your local provider?

  5. Do you think pulmonary rehabilitation is beneficial for patients? In what ways? Or why not?

  6. How easy or difficult is it for you to refer to your local pulmonary rehabilitation provider?

    (for example, is it your role to refer? When is it appropriate to refer COPD patients to pulmonary rehabilitation?)

  7. What motivates you to refer patients to pulmonary rehabilitation?

    (for example, do patients/carers ever ask you about pulmonary rehabilitation? Does the post-pulmonary rehabilitation patient summary motivate you? Are you reminded by prompts or other guidance?)

  8. What do you think stops you from referring patients to pulmonary rehabilitation?

    Imagesa — two images shown to each PHCP, the sequence (A & D) or (B & C) alternated to reflect sex and possible disease severity differences.

  9. If this person was your COPD patient, would you consider discussing pulmonary rehabilitation with them? Why?/Why not?

  10. Is there anything that you think could improve the primary care discussion surrounding pulmonary rehabilitation and/or encourage you to make referrals to pulmonary rehabilitation?

  • ↵a The images are available from the authors on request. COPD = chronic obstructive pulmonary disease.

Box 2.

Topic guide

Photographic images depicting patients with varying COPD severity were used within each interview to illuminate prior experiences and thoughts around PR referral.43 The images are available from the authors on request. Data collection continued until theme saturation.44

Data analysis

An iterative whole team two-stage approach to data analysis was undertaken using rapid qualitative inquiry.45 Rapid qualitative inquiry is helpful for gaining preliminary insight into a complex phenomenon, particularly where little is known and understanding the phenomenon is key.

The transcripts were initially coded to identify themes and patterns within the data. Independent coding by team members occurred following completion of the first five interviews. Emerging themes were then collaboratively discussed and compared to ensure team agreement and the topic guide remained focused. Finally, further independent coding and collaborative team agreement about emergent themes took place again after 14 interviews.

RESULTS

Recruitment

A total of 23 participants initially agreed to participate, with 19 PHCPs from 16 practices across three CCGs (seven from Cambridgeshire and Peterborough, and nine from the West Midlands) agreeing to interviews.

Six participants were recruited through email, six through regional respiratory meeting attendance, and seven from link-tracing sampling.37

Eighteen face-to-face interviews and one telephone interview were conducted between February and November 2017. Interviews lasted 18–63 minutes (mean = 44.7 minutes). Although all participants indicated that they referred patients with COPD to PR, at interview it emerged that two had no knowledge of PR and had never referred, and a further participant no longer reviewed patients with COPD in practice but considered their respiratory knowledge relevant. All three participants’ interview data were included in the analysis as their practice was deemed important to the study. The remainder (n = 16) narratively reported infrequently referring to PR, which differed to what they had documented on the returned self-completed reply slips as detailed in Table 1.

View this table:
  • View inline
  • View popup
Table 1.

Participant characteristics (N = 19)

Themes

Four main themes and eleven subthemes were derived from frequencies and patterns in the interview data.

Figure 1 shows that there is a hierarchical structure to the themes. Awareness and understanding of PR is an important theme because awareness must be present before PHCPs can comment on other aspects. However, where professionals were aware of PR, the characteristics of the PR service, perceptions about patients’ ability to attend/benefit, the referral processes, and personal and collective experiences impacted on the perceived value of PR and subsequent referral behaviour.

Figure 1.
  • Download figure
  • Open in new tab
  • Download powerpoint
Figure 1.

Main themes and subthemes developed from thematic analysis.

The themes are discussed in depth below with quotes that reflect the range of views expressed.

Awareness and understanding about PR

Awareness of the exact nature of PR was variable, with 17 out of 19 reporting some understanding. A small number of PHCPs spoke about PR as a fundamental treatment for patients with COPD: ‘I, 100% feel it is a positive treatment for these patients.’(practice nurse [PN]1)

All participants but one (advanced nurse practitioner [ANP]3) reported that they thought PR to be an intervention that incorporated exercise for people with COPD, thereby helping patients to manage their breathlessness: ‘Is all about understanding why they get breathless and how to manage that breathlessness.’(PN2)

This awareness was gained from many sources including postgraduate respiratory training and respiratory interested/specialist secondary care colleagues.

One GP shared an anecdote, which he described had positively influenced his view of PR and current practice: ‘ ... [the GP] asked the consultant “which inhaler is best? ” The respiratory consultant replied “pulmonary rehabilitation”.’(GP5)

Local CCG interventions such as quality improvement programmes improved awareness of PR: ‘We did it as part of the PDMA, Practice Development and Membership Agreement.’(GP3)

PNs appeared to be more aware of PR than other PHCPs, yet few were able to fully describe it. Four out of seven PNs and one out of four ANPs described having observed PR, yet one went on to say: ‘I know what pulmonary rehabilitation is, but what is it really?’(ANP2)

One participant had previously been involved with PR delivery and advocated its benefits, but reported referring few patients: ‘It’s beneficial for the right patient; expectations are different to reality.’(PN1)

This participant conveyed a perception that was echoed by several other PHCPs, suggesting that knowledge of PR could itself impede referral.

Overall, irrespective of level of awareness, most PHCPs underestimated the value of PR in comparison with other interventions: ‘Quite often they’re already on triple-inhaled therapy and you’re sort of thinking, what else might help and that’s when it perhaps comes to mind that maybe pulmonary rehabilitation might be suitable.’(GP6)

Despite the sample being composed of PHCPs with some interest in COPD, the awareness and appreciation of the relative value of PR among all treatments was relatively poor.

Characteristics of pulmonary rehabilitation services

Both general features of PR and local PR service providers appeared to influence referral. An important characteristic that PHCPs frequently described as impeding the ‘promotion’ of PR, is the name itself: ‘Pulmonary rehabilitation, which in itself sounds horrendously frightening.’(ANP4)

Consequently, PHCPs often pre-empted negative patient responses to PR and used alternative terms, such as ‘exercise’ or ‘management’ when describing PR to patients, while others described avoiding the term ‘exercise’ completely: ‘I try not to term it as an exercise programme; I tend to explain it as a treatment programme.’(PN1)

And for some, particularly nurses, greater concern existed about offering PR to patients who they perceived not to benefit: ‘If you raise patients’ hopes and say offer it … it could make the depression worse.’(ANP4)

PHCPs rarely described direct PR benefits, such as improvements in emotional function as highlighted in Box 1.

Where PHCPs knew about PR, they often described articulating it as difficult: ‘I can’t always put into words, you know, summarise what the benefits are. You know, sort of in two sentences.’(PN2)

This was linked to frequent reports of a lack of local practice PR promotional material.

The three subthemes relative to this main theme are now discussed in further detail.

Provider engagement

Almost all participants described little engagement from PR providers, mentioning details such as wait time and timing of provision to be frequently unknown: ‘ There’s a definite lack of understanding about when it runs, how often it runs, who the lead is, what sort of exercises they do.’(PN1)

Additionally, very few PHCPs knew if, or when, patients who had attended PR could re-attend.

Quality of local service including personal perceptions

Wide variations in exercise type offered in PR programmes were described, including tai chi and practice-based chair exercises for general populations. Variation across PR programmes was more prevalent in the West Midlands than in Cambridgeshire and Peterborough; however, differences in content appeared not to influence referral. Instead, factors, such as intervention venue, ease of referral, patient acceptance, and perception of PR providers were considered by PHCP referrers: ‘At the minute I’m probably saying … Because it’s local and I think they might buy it. The referral process is really easy, I send an internal message — and it’s done.’(PN7)

Location

Participants frequently cited location as a barrier to PR referral yet it is not clear if this was voiced by patients or anticipated by PHCPs: ‘It’s two buses from [name of venue] and they won’t go for two buses.’(PN4)

However, not all PHCPs knew the venue, or were able to confirm to patients where providers might send them: ‘ We can request, but it doesn’t always end up being where we say it is, and where we’ve asked it to be.’(GP4)

Indeed, the emergence of practice-based group exercises appeared to be a solution to this problem, particularly in the West Midlands.

Perceptions about patients

A further important theme that emerged from the data is the PHCPs’ perceptions of patients’ physical and/or psychological ability to benefit from PR. This theme comprises four subthemes.

Clinical characteristics

PHCPs largely described PR as an intervention for patients with defined clinical characteristics, particularly for deteriorating patients, measured by breathless, declining FEV1, or increasing exacerbation frequency. A frequently mentioned PR referral requirement was patients having to meet Medical Research Council (MRC) breathlessness score ≥3:5 ‘I tend to base it on their MRC scores, so the people who are scoring MRC 3, 4, or 5.’(GP6)

Nevertheless, assessment of suitability was often subjective, as demonstrated by the photographic images used. Most assumed the patient in image B who was standing upright dressed in a suit did not require referral based on his appearance, and without need for clinical assessment: ‘He’s got his walking boots on, I mean he’s quite fit.’(PN7)

On the other hand, there was reluctance about referring patients who used oxygen. This was partly based on preconceptions around potential benefit: ‘I would worry about whether she could physically participate … or actually benefit.’ [referring to image C of a woman using oxygen](ANP2)

Others assumed that patients on oxygen would have already been offered PR because of likely contact with secondary care, so they would be less likely to offer PR.

Social characteristics

PHCPs frequently described characteristics, such as social isolation and employment as barriers to attendance: ‘I think the majority of patients who say no to me, it’s because of that anxiety they’ve got about going out, I think it’s just easier to say no.’(ANP4)

Yet, paradoxically PHCPs often viewed PR positively in terms of potential social interactions, particularly for those considered socially isolated.

Patients in work were described as being inhibited by PR programme timing; however, PHCPs frequently considered them as being too well to benefit: ‘The biggest thing is when they are still in work and it’s going to jeopardise their work situation to take time off to go, and they’re not severe yet.’(ANP1)

A small number of PHCPs also described current smokers as inappropriate for PR: ‘If a patient won’t give up smoking almost what’s the point in doing it.’(practice manager [PM]1)

Motivation

PHCPs frequently reported that patients are not motivated to attend and described having to ‘push’ and ‘nag’ patients: ‘They won’t go to PR.’(PN7) ‘Patients don’t want it.’(PN5) ‘The main hurdle is convincing people to go.’(GP5)

This described lack of patient motivation can lead to variation in PR introduction approaches: ‘You sort of need to trickle feed it.’(PN4)

Relationships

Relationships between PHCPs and patients were seen as important, but could result in differing patient reactions. Long-term contact with PHCPs resulted in trust for some: ‘I’ve seen patients for many many years … they’ve confidence in you and will say “well, if you think that’s its good for me I will go to it”.’(PN3)

In contrast, others found familiarity a barrier: ‘Not again, don’t start, not again.’(PN7)

The relationship also impacted on PHCPs’ reactions, with some describing caring for patients with COPD as challenging, captured by the term ‘heartsink patients’ (PN2 and GP5).

Working environment and referral processes

This final theme included four subthemes. Discussions about PR referral were largely seen as most appropriately undertaken by practice nurses, at COPD annual review: ‘When they’re doing the routine review … that’s probably the ideal time.’(GP5)

GPs largely reported referring infrequently: ‘I can’t remember the last time I referred anyone.’(GP3)

Few PHCPs considered PR discussion appropriate in an acute exacerbation consultation, given clinical priority and consultation time constraints: ‘It doesn’t form any part of my acutely unwell consultation.’(GP4)

However, post-exacerbation review in primary care is an emerging practice, particularly in the West Midlands. This was described as offering an additional referral opportunity: ‘It may get better now we’re trying to do these post-exacerbation reviews because then you’ve got that second opportunity to look and say.’(GP6)

Consultation time and priorities

PHCPs often reported that clinical time constraints inhibit PR referral and described rationalising organisational priorities: ‘ The biggest barrier to any referral is time … [on PR] it’s almost like an optional thing you can consider, not something you have to do.’(ANP2) ‘ There are lots of calls on prioritisation.’(GP2)

Ease of referral

Most PHCPs knew how to refer, describing the process as straightforward, despite some variations. The PR referral on COPD templates was frequently described as a valuable reminder: ‘ The template would always prompt.’(healthcare assistant [HCA]1)

Patient and provider feedback

Positive patient feedback motivated PHCPs to discuss PR with subsequent patients with COPD: ‘I have seen patients who … have been suitable for PR but haven’t agreed to be referred in the past … I would mention … the experience of other patients … have found it helpful so that might motivate them to kind of agree.’(GP1)

However, PR provider feedback was largely described as very poor: ‘I don’t know what’s happened to my patients that I’ve referred, whether they’ve actually gone, what their outcome is.’(ANP4)

Additionally, where providers had deemed referred patients ineligible for PR, it appeared to be patients rather than providers that fed that information back to the PHCPs. PHCPs described feeling frustrated in these circumstances, demotivating them from future referrals: ‘I was a bit annoyed because I felt quite strongly that he would benefit.’(PN3)

Public awareness of PR appeared very low, with only two out of 19 participants reporting patients/carers to have asked about PR. PR awareness within some general practices also appeared to be low: ‘It’s very rarely talked about, I’ve never heard it mentioned.’(ANP3)

Practice incentives

Financial incentives appeared to differ between the locations and influence PHCPs’ views. In Cambridgeshire and Peterborough there was no financial incentive for PR referral: ‘I’m not saying it should be but if pulmonary COPD was a QOF priority, money attached, we would do it.’(GP2)

In the West Midlands there were financial incentives, which were deemed as helpful in terms of reminders: ‘Maybe triggers other people’s brains … it’s part of a thing you need to do.’(GP6)

They were also deemed helpful in raising the profile of PR and authorising referrals: ‘It makes it okay.’(GP6)

Suggested enablers

All participants were invited to suggest possible enablers to PR referral. Box 3 summarises the 12 suggestions that were proposed and their alignment to the four main themes.

EnablersThemes
Educate PHCP staff on PRAwareness and understanding of PR
Changing the PR nameCharacteristics of PR services
Monthly feedback from PR providers
PR closer to practice/in practice
Patient self-referral
Directly invite eligible patients
PHCP use of motivational interviewingPerceptions about patients
Patient to watch video
Post exacerbation follow-upWorking environment and referral
Prompts and/or reminders to staff
Additional practice funding/incentive PR posters in practices
  • PHCP = primary healthcare practitioner. PR = pulmonary rehabilitation.

Box 3.

Suggested enablers and relation to main themes

DISCUSSION

Summary

This is the first in depth qualitative study to investigate a range of PHCPs’ experiences of referring or considering PR referral for patients with COPD in England. It highlights a hierarchical approach to PR discussion and referral by PHCPs that has not previously been reported. This is dominated initially by PHCPs’ individual awareness and understanding of PR, which is subsequently strongly influenced by three further factors: characteristics of PR services, perceptions about patient motivation and characteristics, and the working environment and referral process.

PR referral is a highly complex phenomenon, which was influenced by the PHCPs’ knowledge and awareness of the benefits of PR, and contextual factors such as perceived quality of the PR service, engagement with providers, consultation time, referral prompts, ease of referral, PHCP–patient relationship, perceptions of the patient’s needs and abilities, and practical considerations, such as PR accessibility. While detailed awareness of the clinical benefits of PR was generally low, it is the combined total of these factors that appeared to greatly inhibit PR referral by PHCPs. As such, increasing PHCPs’ knowledge alone will not necessarily translate to increased referral rates; indeed, some PR knowledge appears to impede referral as knowledgeable PHCPs analyse patient assessment and suitability in greater detail.

PHCPs described referral to be considered largely at COPD annual review, yet frequent subjective evaluation of patient characteristics rather than clinical assessment dominated decisions. This subjective assessment led to referral opportunities being considerably narrowed.

PHCPs frequently reported that patients were unmotivated and perceived patient-based difficulties in attending PR as important barriers to referral. In response, a commonly used lever was to offer PR when COPD symptom burden, such as exacerbation rate and dyspnoea is increasing and where pharmacological treatments have been maximised. PR was therefore frequently viewed as an end of the road treatment at a time where patients with COPD often have high symptom burden. PHCPs were aware of this and not always certain how best to support patients, although described introducing PR gradually as one approach.

Relationships with PR providers and environmental factors, such as having the time to refer and practical accessibility of PR service for patients influenced referral behaviours. PHCPs often described a lack of familiarity with their local PR providers contributing to lower referral. Indeed, the most frequent suggested enablers to improve PR referral (Box 3) were related to PR providers.

While it was not intended to make comparisons between the geographical areas, some differences were noted. In some West Midlands practices, PR incentive funding appeared to increase PHCPs’ PR awareness and increased reported referral rates; however, higher non-elective COPD emergency admissions in this location31 may also contribute.

Strengths and limitations

This study was able to recruit a wide range of general practice-based participants, with knowledge of PR, in all but two out of 19 participants. Gaining insight into PHCPs’ real experiences of current PR referral barriers and enablers is important.

Only PHCPs who had an interest in PR and/or COPD may have agreed to take part and had the study been undertaken in different geographical regions, findings may have differed. All practice nurses and one of the GPs interviewed had a post-qualifying respiratory qualification, which may not be reflective of the general practice workforce. Despite this, the awareness and appreciation of the relative value of PR among treatment options for COPD was limited.

One of the authors is an experienced respiratory specialist nurse, whose experiences may have influenced data collection and analysis.38

Comparison with existing literature

PHCPs’ knowledge of PR has previously been reported as being low.23,26 While general HCPs’ awareness may have increased since the implementation of secondary care COPD discharge bundles, PR referral and uptake remains low.46,47

The current study found that most PHCPs had a general PR awareness, but few had detailed PR knowledge. While knowledge is a key factor to referral, this is highly influenced by additional factors that must be addressed if PR referral rates are to increase and PHCPs are to improve its ‘promotion’.

An early qualitative study24 and a practice-based service evaluation13 postulated that PHCPs’ attitudes may contribute to low PR referral numbers, a factor reported by patients themselves.19,21 Conversely, the current study found that PHCPs frequently described patients as having little motivation for PR, as has also been reported elsewhere.24,26 Participants described maximal pharmacological therapy and worsening COPD symptoms as opportunities for motivating patients to accept PR.

This approach has important implications. First, patients with high COPD symptoms live with high levels of fear and panic,20,48 and have concerns about functional abilities, including beliefs that shortness of breath is detrimental to lung capacity.20 These are highly likely to influence patients’ referral acceptance and may explain ‘low motivation’. Second, many patients with COPD are frequently inappropriately prescribed triple therapy. Referring patients earlier to PR when their symptom burden is less may increase patient acceptance and may reduce prescribing behaviours.

Patients have reported that key factors to referral acceptance are their understanding of the content and benefits of participation in PR programmes.17 The findings of this study highlight that PHCPs struggle to articulate both content and clinical benefit of PR. HCPs are highly influential in a patient’s acceptance of PR referral,15–18 yet those interviewed had little awareness of this and subsequently no plans to address it.

Financial incentives appeared to facilitate the likelihood of referral in this study and elsewhere.49 It is likely that the implementation of financial reward for PR referral instilled by the new GP contract changes in England and Wales50 will yield some positive changes. However, it is important to acknowledge that increased GP PR referral rates do not necessarily translate to greater PR completion, as has been reported by Hogg et al.51 As such, adopting a system-based approach at the point of referral that incorporates the three key parties (PHCP, patient, and PR provider) is likely to be more successful in overcoming the multiple barriers than isolated interventions.

Implications for research and practice

PR is a nationally and internationally recommended intervention for patients with COPD, yet it remains poorly accessed. Primary care consultations for COPD are increasing, as is the COPD population. Increasing PR referral and likely uptake will reduce patient symptom burden and potentially reduce primary care contacts.

The PR referrer is very important to referral acceptance, uptake, and completion. PHCPs reported large numbers of barriers and very few enablers to PR referral. This appears to indicate that a multipronged approach to enhancing referral is likely to be valuable. This goes beyond education and awareness raising for PHCPs, and includes engagement of PR providers and system-level changes. Identifying the frequency and generalisability of these findings are important next steps. These current findings have been mapped to a national PHCPs survey, for which data has been collected and analysis is underway.

Acknowledgments

The authors thank all participating GPs, nurses, and the healthcare assistant for giving up their time, providing the data, and contributing to this study.

Notes

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Ethical approval

This research was approved by the NHS Health Research Authority (reference number: 17/HRA/0008) (Integrated Research Application System project ID: 213367) (protocol number: RG_16-032) and sponsored by the University of Birmingham.

Provenance

Freely submitted; externally peer reviewed.

Competing interests

The authors have declared no competing interests.

Discuss this article

Contribute and read comments about this article: bjgp.org/letters

  • Received April 29, 2019.
  • Revision requested June 7, 2019.
  • Accepted July 23, 2019.
  • © British Journal of General Practice 2020

REFERENCES

  1. 1.↵
    1. Department of Health and Social Care.
    An outcomes strategy for COPD and asthma: NHS companion document2011https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/216531/dh_134001.pdf (accessed 3 Jan 2020).
  2. 2.↵
    1. NHS Digital.
    Hospital admitted patient care activity, 2017–18.https://digital.nhs.uk/data-and-information/publications/statistical/hospital-admitted-patient-care-activity/2017-18 (accessed 3 Jan 2020).
  3. 3.↵
    1. Punekar YS,
    2. Shukla A,
    3. Mullerova H
    COPD management costs according to the frequency of COPD exacerbations in UK primary careInt J Chron Obstruct Pulmon Dis201496573
    OpenUrl
  4. 4.↵
    1. Global Initiative for Chronic Obstructive Lung Disease.
    Global strategy for the diagnosis, management, and prevention of chronic obstructive pulmonary disease: 2019 report2019https://goldcopd.org/wp-content/uploads/2017/11/GOLD-2018-v6.0-FINAL-revised-20-Nov_WMS.pdf (accessed 3 Jan 2020).
  5. 5.↵
    1. National Institute for Health and Care Excellence.
    Chronic obstructive pulmonary disease in over 16s: diagnosis and management. NG1152019https://www.nice.org.uk/guidance/ng115 (accessed 3 Jan 2020).
  6. 6.↵
    1. Bolton CE,
    2. Bevan-Smith EF,
    3. Blakey JD,
    4. et al.
    British Thoracic Society guideline on pulmonary rehabilitation in adults: accredited by NICEThorax201368suppl 2ii1ii30
    OpenUrlFREE Full Text
  7. 7.↵
    1. Royal College of Physicians (RCP),
    2. British Thoracic Society
    National COPD Audit Programme. COPD: who cares matters. National clinical audit report, February 2015LondonRCP2015
  8. 8.↵
    1. Meis JJ,
    2. Bosma CB,
    3. Spruit MA,
    4. et al.
    A qualitative assessment of COPD patients’ experiences of pulmonary rehabilitation and guidance by healthcare professionalsRespir Med20141083500510
    OpenUrl
  9. 9.
    1. Zoeckler N,
    2. Kenn K,
    3. Kuehl K,
    4. et al.
    Illness perceptions predict exercise capacity and psychological well-being after pulmonary rehabilitation in COPD patientsJ Psychosom Res2014762146151
    OpenUrlCrossRefPubMed
  10. 10.↵
    1. McCarthy B,
    2. Casey D,
    3. Devane D,
    4. et al.
    Pulmonary rehabilitation for chronic obstructive pulmonary disease.Cochrane Database Syst Rev20152CD003793DOI: https://doi.org/10.1002/14651858.CD003793.pub3.
  11. 11.↵
    1. Puhan MA,
    2. Gimeno-Santos E,
    3. Cates CJ,
    4. Troosters T
    Pulmonary rehabilitation following exacerbations of chronic obstructive pulmonary disease.Cochrane Database Syst Rev201612CD005305DOI: https://doi.org/10.1002/14651858.CD005305.pub4.
  12. 12.↵
    1. Royal College of Physicians (RCP),
    2. British Thoracic Society
    Pulmonary rehabilitation: steps to breathe better. National clinical audit report, February 2016LondonRCP2016
  13. 13.↵
    1. Foster F,
    2. Piggott R,
    3. Riley L,
    4. Beech R
    Working with primary care clinicians and patients to introduce strategies for increasing referrals for pulmonary rehabilitationPrim Health Care Res Dev2016173226237
    OpenUrl
  14. 14.↵
    1. Moore E,
    2. Newson R,
    3. Joshi M,
    4. et al.
    Effects of pulmonary rehabilitation on exacerbation number and severity in people with COPD: an historical cohort study using electronic health recordsChest2017152611881202
    OpenUrl
  15. 15.↵
    1. Moore L,
    2. Hogg L,
    3. White P
    Acceptability and feasibility of pulmonary rehabilitation for COPD: a community qualitative studyPrim Care Respir J2012214419424
    OpenUrlCrossRefPubMed
  16. 16.
    1. Fischer MJ,
    2. Scharloo M,
    3. Abbink JJ,
    4. et al.
    Participation and drop-out in pulmonary rehabilitation: a qualitative analysis of the patient’s perspectiveClin Rehabil2007213212221
    OpenUrlCrossRefPubMed
  17. 17.↵
    1. Keating A,
    2. Lee A,
    3. Holland AE
    What prevents people with chronic obstructive pulmonary disease from attending pulmonary rehabilitation? A systematic reviewChron Respir Dis2011828999
    OpenUrlCrossRefPubMed
  18. 18.↵
    1. Arnold E,
    2. Bruton A,
    3. Ellis-Hill C
    Adherence to pulmonary rehabilitation: a qualitative studyRespir Med20061001017161723
    OpenUrlCrossRefPubMed
  19. 19.↵
    1. Harrison SL,
    2. Robertson N,
    3. Apps L,
    4. et al.
    ‘We are not worthy’: understanding why patients decline pulmonary rehabilitation following an acute exacerbation of COPDDisabil Rehabil2015379750756
    OpenUrlCrossRefPubMed
  20. 20.↵
    1. Guo SE,
    2. Bruce A
    Improving understanding of and adherence to pulmonary rehabilitation in patients with COPD: a qualitative inquiry of patient and health professional perspectivesPLoS One2014910e110835
    OpenUrl
  21. 21.↵
    1. Harris D,
    2. Hayter M,
    3. Allender S
    Improving the uptake of pulmonary rehabilitation in patients with COPD: qualitative study of experiences and attitudes.Br J Gen Pract2008http://doi.org/10.3399/bjgp08X342363
  22. 22.
    1. Bulley C,
    2. Donaghy M,
    3. Howden S,
    4. et al.
    A prospective qualitative exploration of views about attending pulmonary rehabilitationPhysiother Res Int2009143181192
    OpenUrlCrossRefPubMed
  23. 23.↵
    1. Cox NS,
    2. Oliveira CC,
    3. Lahham A,
    4. Holland AE
    Pulmonary rehabilitation referral and participation are commonly influenced by environment, knowledge, and beliefs about consequences: a systematic review using the Theoretical Domains FrameworkJ Physiother20176328493
    OpenUrl
  24. 24.↵
    1. Harris D,
    2. Hayter M,
    3. Allender S
    Factors affecting the offer of pulmonary rehabilitation to patients with chronic obstructive pulmonary disease by primary care professionals: a qualitative studyPrimary Health Care Research & Development200894280290
    OpenUrl
  25. 25.↵
    1. Royal College of Physicians of London (RCP),
    2. British Thoracic Society, British Lung Foundation
    Report of the National Chronic Obstructive Pulmonary Disease Audit 2008.LondonRCP2008
  26. 26.↵
    1. Milner SC,
    2. Boruff JT,
    3. Beaurepaire C,
    4. et al.
    Rate of, and barriers and enablers to, pulmonary rehabilitation referral in COPD: a systematic scoping reviewRespir Med2018137103114
    OpenUrlCrossRefPubMed
  27. 27.↵
    1. Johnston KN,
    2. Young M,
    3. Grimmer-Somers KA,
    4. et al.
    Why are some evidence-based care recommendations in chronic obstructive pulmonary disease better implemented than others? Perspectives of medical practitionersInt J Chron Obstruct Pulmon Dis20116659667
    OpenUrlPubMed
  28. 28.
    1. Johnston KN,
    2. Young M,
    3. Grimmer KA,
    4. et al.
    Barriers to, and facilitators for, referral to pulmonary rehabilitation in COPD patients from the perspective of Australian general practitioners: a qualitative studyPrim Care Respir J2013223319324
    OpenUrlCrossRefPubMed
  29. 29.↵
    1. Johnston K,
    2. Grimmer-Somers K,
    3. Young M,
    4. et al.
    Which chronic obstructive pulmonary disease care recommendations have low implementation and why? A pilot studyBMC Res Notes20125652
    OpenUrlCrossRefPubMed
  30. 30.↵
    1. NHS Digital
    Quality and Outcomes Framework (QOF) — 2015–16. This is part of Quality and Outcomes Framework, Achievement, prevalence and exceptions data.QOF 2015–16: Prevalence achievements and exceptions at CCG level v2. 2016. https://digital.nhs.uk/data-and-information/publications/statistical/quality-and-outcomes-framework-achievement-prevalence-and-exceptions-data/quality-and-outcomes-framework-qof-2015-16 (accessed 9 Jan 2020).
  31. 31.↵
    1. Public Health England.
    The 2nd Atlas of variation in risk factors and healthcare resporatory disease in England.Map 11a: Variation in rate of emergency admissions to hospital for COPD per population by CCG. http://tools.england.nhs.uk/images/RespiratoryAtlas/atlas.html (accessed 9 Jan 2020).
  32. 32.↵
    1. Birmingham South Central Clinical Commissioning Group.
    Birmingham South Central CCG EDS2 Report2017https://www.birminghamandsolihullccg.nhs.uk/about-us/publications/equality-diversity-inclusion/28-equality-delivery-system-update-birmingham-south-central-ccg-2017/file (accessed 3 Jan 2020).
  33. 33.↵
    1. Department for Communities and Local Government.
    The English indices of deprivation 2015. Statistical release 30 September 20152015https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/465791/English_Indices_of_Deprivation_2015_-_Statistical_Release.pdf (accessed 3 Jan 2020).
  34. 34.↵
    1. Mukhtar TK,
    2. Bankhead C,
    3. Stevens S,
    4. et al.
    Factors associated with consultation rates in general practice in England, 2013–2014: a cross-sectional study.Br J Gen Pract2018DOI: https://doi.org/10.3399/bjgp18X695981.
  35. 35.↵
    1. Office for National Statistics.
    Clinical commissioning group population estimates (National Statistics), mid-20172017https://www.ons.gov.uk/peoplepopulationandcommunity/populationandmigration/populationestimates/datasets/clinicalcommissioninggroupmidyearpopulationestimates (accessed 3 Jan 2020).
  36. 36.↵
    1. Lavrakas P
    1. Dryver A,
    2. Lavrakas P
    Adaptive sampling methods.Lavrakas PEncyclopedia of survey research methodsThousand Oaks CASage Publications200856
  37. 37.↵
    1. Thompson SK
    Sampling3rd ednHoboken, NJJohn Wiley & Sons2012
  38. 38.↵
    1. Fowler FJ
    Survey research methods5th ednLondonSAGE Publications2014
  39. 39.↵
    1. Ritchie J,
    2. Lewis J,
    3. McNaughton Nicholls C,
    4. Ormston R
    1. Yeo A,
    2. Legard R,
    3. Keegan J,
    4. et al.
    In-depth interviewsRitchie J, Lewis J, McNaughton Nicholls C, Ormston RQualitative research practice: a guide for social science students and researchers2nd ednLondonSAGE Publications2013177210
  40. 40.↵
    1. Irvine A
    Realities Tool Kit #14: Using phone interviews2010http://eprints.ncrm.ac.uk/1576/1/14-toolkit-phone-interviews.pdf (accessed 3 Jan 2020).
  41. 41.↵
    1. Hagens V,
    2. Dobrow MJ,
    3. Chafe R
    Interviewee transcript review: assessing the impact on qualitative researchBMC Med Res Methodol2009947
    OpenUrlPubMed
  42. 42.↵
    1. Michie S,
    2. Atkins L,
    3. West R
    The behaviour change wheel: a guide to designing interventions.SuttonSilverback2014
  43. 43.↵
    1. Gong F,
    2. Castaneda D,
    3. Zhang X,
    4. et al.
    Using the associative imagery technique in qualitative health research: the experiences of homecare workers and consumersQual Health Res2012221014141424
    OpenUrlCrossRefPubMed
  44. 44.↵
    1. Grady MP
    Qualitative and action research: a practitioner handbook.Bloomington, INPhi Delta Kappa Educational Foundation1998
  45. 45.↵
    1. Beebe J
    Rapid qualitative inquiry: a field guide to team-based assessment.2nd edn.LondonRowman & Littlefield2014
  46. 46.↵
    1. Jones SE,
    2. Green SA,
    3. Clark AL,
    4. et al.
    Pulmonary rehabilitation following hospitalisation for acute exacerbation of COPD: referrals, uptake and adherenceThorax2014692181182
    OpenUrlAbstract/FREE Full Text
  47. 47.↵
    1. Migone C,
    2. O’Conor M,
    3. Kelly E,
    4. McDonnell TJ
    Patients hospitalised with an acute excerbation of COPD: is there a need for a discharge bundle of care?Ir Med J20151089273275
    OpenUrl
  48. 48.↵
    1. Lewis A,
    2. Bruton A,
    3. Donovan-Hall M
    Uncertainty prior to pulmonary rehabilitation in primary care: a phenomenological qualitative study in patients with chronic obstructive pulmonary diseaseChron Respir Dis2014113173180
    OpenUrlCrossRefPubMed
  49. 49.↵
    1. Hull S,
    2. Mathur R,
    3. Lloyd-Owen S,
    4. et al.
    Improving outcomes for people with COPD by developing networks of general practices: evaluation of a quality improvement project in east LondonNPJ Prim Care Respir Med20142414082
    OpenUrl
  50. 50.↵
    1. NHS England, British Medical Association.
    Investment and evolution: a five-year framework for GP contract reform to implement the NHS Long Term Plan2019https://www.england.nhs.uk/wp-content/uploads/2019/01/gp-contract-2019.pdf (accessed 3 Jan 2020).
  51. 51.↵
    1. Hogg L,
    2. Garrod R,
    3. Thorton H,
    4. et al.
    Effectiveness, attendance, and completion of an integrated, system-wide pulmonary rehabilitation service for COPD: prospective observational studyCOPD201295546554
    OpenUrlCrossRef
Back to top
Previous ArticleNext Article

In this issue

British Journal of General Practice: 70 (693)
British Journal of General Practice
Vol. 70, Issue 693
April 2020
  • Table of Contents
  • Index by author
Download PDF
Download PowerPoint
Email Article

Thank you for recommending British Journal of General Practice.

NOTE: We only request your email address so that the person to whom you are recommending the page knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Enter multiple addresses on separate lines or separate them with commas.
Referral of patients with chronic obstructive pulmonary disease to pulmonary rehabilitation: a qualitative study of barriers and enablers for primary healthcare practitioners
(Your Name) has forwarded a page to you from British Journal of General Practice
(Your Name) thought you would like to see this page from British Journal of General Practice.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Citation Tools
Referral of patients with chronic obstructive pulmonary disease to pulmonary rehabilitation: a qualitative study of barriers and enablers for primary healthcare practitioners
Jane S Watson, Peymane Adab, Rachel E Jordan, Alexandra Enocson, Sheila Greenfield
British Journal of General Practice 2020; 70 (693): e274-e284. DOI: 10.3399/bjgp20X708101

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero

Share
Referral of patients with chronic obstructive pulmonary disease to pulmonary rehabilitation: a qualitative study of barriers and enablers for primary healthcare practitioners
Jane S Watson, Peymane Adab, Rachel E Jordan, Alexandra Enocson, Sheila Greenfield
British Journal of General Practice 2020; 70 (693): e274-e284. DOI: 10.3399/bjgp20X708101
del.icio.us logo Digg logo Reddit logo Twitter logo CiteULike logo Facebook logo Google logo Mendeley logo
  • Tweet Widget
  • Facebook Like
  • Google Plus One
  • Mendeley logo Mendeley

Jump to section

  • Top
  • Article
    • Abstract
    • INTRODUCTION
    • METHOD
    • RESULTS
    • DISCUSSION
    • Acknowledgments
    • Notes
    • REFERENCES
  • Figures & Data
  • Info
  • eLetters
  • PDF

Keywords

  • COPD
  • pulmonary disease, chronic obstructive
  • pulmonary rehabilitation
  • referral

More in this TOC Section

  • Improving the diagnostic process for patients with possible bladder and kidney cancer: a mixed-methods study to identify potential missed diagnostic opportunities
  • National Cancer Diagnosis Audits for England 2018 versus 2014: a comparative analysis
  • Symptom appraisal and help seeking in males with symptoms of possible prostate cancer: a qualitative study with an ethnically diverse sample in London
Show more Research

Related Articles

Cited By...

Intended for Healthcare Professionals

BJGP Life

BJGP Open

 

 
 

British Journal of General Practice

NAVIGATE

  • Home
  • Current Issue
  • All Issues
  • Online First
  • Authors & reviewers

RCGP

  • BJGP for RCGP members
  • BJGP Open
  • RCGP eLearning
  • InnovAiT Journal
  • Jobs and careers

MY ACCOUNT

  • RCGP members' login
  • Subscriber login
  • Activate subscription
  • Terms and conditions

NEWS AND UPDATES

  • About BJGP
  • Alerts
  • RSS feeds
  • Facebook
  • Twitter

AUTHORS & REVIEWERS

  • Submit an article
  • Writing for BJGP: research
  • Writing for BJGP: other sections
  • BJGP editorial process & policies
  • BJGP ethical guidelines
  • Peer review for BJGP

CUSTOMER SERVICES

  • Advertising
  • Contact subscription agent
  • Copyright
  • Librarian information

CONTRIBUTE

  • BJGP Life
  • eLetters
  • Feedback

CONTACT US

BJGP Journal Office
RCGP
30 Euston Square
London NW1 2FB
Tel: +44 (0)20 3188 7400
Email: journal@rcgp.org.uk

British Journal of General Practice is an editorially-independent publication of the Royal College of General Practitioners
© 2023 British Journal of General Practice

Print ISSN: 0960-1643
Online ISSN: 1478-5242