You are here

  • Home
  • Archive
  • Volume 8, Issue 3
  • Use of the nominal group technique to identify stakeholder priorities and inform survey development: an example with informal caregivers of people with scleroderma

Article Text

Article menu

Download PDFPDF

XML
Research methods
Research
Use of the nominal group technique to identify stakeholder priorities and inform survey development: an example with informal caregivers of people with scleroderma
  1. Danielle B Rice1,2,
  2. Mara Cañedo-Ayala1,
  3. Kimberly A Turner1,
  4. Stephanie T Gumuchian3,
  5. Vanessa L Malcarne4,5,
  6. Mariët Hagedoorn6,
  7. Brett D Thombs1,2,3,7,8,9
  8. Scleroderma Caregiver Advisory Team
    1. 1 Lady Davis Institute for Medical Research, Jewish General Hospital, Montreal, Quebec, Canada
    2. 2 Department of Psychology, McGill University, Montreal, Quebec, Canada
    3. 3 Department of Educational and Counselling Psychology, McGill University, Montreal, Quebec, Canada
    4. 4 Department of Psychology, San Diego State University, San Diego, California, USA
    5. 5 San Diego Joint Doctoral Program in Clinical Psychology, San Diego State University and the University of California, San Diego, California, USA
    6. 6 University Medical Center Groningen, University of Groningen, Groningen, The Netherlands
    7. 7 Department of Psychiatry, McGill University, Montreal, Quebec, Canada
    8. 8 Department of Epidemiology, Biostatistics and Occupational Health, McGill University, Montreal, Quebec, Canada
    9. 9 Department of Medicine, McGill University, Montreal, Quebec, Canada
    1. Correspondence to Dr Brett D Thombs; brett.thombs{at}mcgill.ca

    Abstract

    Objectives The nominal group technique (NGT) allows stakeholders to directly generate items for needs assessment surveys. The objective was to demonstrate the use of NGT discussions to develop survey items on (1) challenges experienced by informal caregivers of people living with systemic sclerosis (SSc) and (2) preferences for support services.

    Design Three NGT groups were conducted. In each group, participants generated lists of challenges and preferred formats for support services. Participants shared items, and a master list was compiled, then reviewed by participants to remove or merge overlapping items. Once a final list of items was generated, participants independently rated challenges on a scale from 1 (not at all important) to 10 (extremely important) and support services on a scale from 1 (not at all likely to use) to 10 (very likely to use). Lists generated in the NGT discussions were subsequently reviewed and integrated into a single list by research team members.

    Setting SSc patient conferences held in the USA and Canada.

    Participants Informal caregivers who previously or currently were providing care for a family member or friend with SSc.

    Results A total of six men and seven women participated in the NGT discussions. Mean age was 59.8 years (SD=12.6). Participants provided care for a partner (n=8), parent (n=1), child (n=2) or friend (n=2). A list of 61 unique challenges was generated with challenges related to gaps in information, resources and support needs identified most frequently. A list of 18 unique support services was generated; most involved online or in-person delivery of emotional support and educational material about SSc.

    Conclusions The NGT was an efficient method for obtaining survey items directly from SSc caregivers on important challenges and preferences for support services.

    • systemic sclerosis
    • scleroderma
    • nominal group technique
    • caregiver

    This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/

    https://doi.org/10.1136/bmjopen-2017-019726

    Statistics from Altmetric.com

      Request Permissions

      If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

      Strengths and limitations of this study

      • This is the first study to use the nominal group technique to investigate the experiences of caregivers of people living with scleroderma.

      • Participating caregivers were asked to deliver answers in a round-robin format, which gave all participants an equal opportunity to describe the challenges they face and their support service needs and preferences.

      • A Scleroderma Caregiver Advisory Team, comprising caregivers of people living with scleroderma, was involved at each stage of the study.

      • Participants were recruited from scleroderma national patient conferences, which may limit the generalisability of findings.

      • Findings will need to be verified with a larger number of caregivers in survey format due to the relatively small number of participants.

      Introduction  

      Surveys provide a feasible method for gathering and prioritising input from large numbers of stakeholders in order to inform programme development.1 A number of approaches can be used to develop survey items for the purpose of needs prioritisation. Examples include adapting a pre-existing questionnaire originally designed for use with other groups, gathering expert opinions and using theoretical definitions of a construct to guide item generation.2–5 These approaches do not explicitly integrate perspectives of stakeholders, however, and risk identification of programme goals that may not be ideally aligned with the needs of the target group.6 Qualitative focus groups and individual patient interviews may also be used to generate items. These methods, however, can be time-intensive and resource-intensive and may overly emphasise input from some participants and not others.7 Furthermore, they require researchers to translate identified themes into survey items, but do not directly generate survey items.

      The nominal group technique (NGT) is an approach that can be used to structure group discussions in a way that allows stakeholders to directly generate items for a needs assessment survey. The NGT method facilitates generation of survey items in a straightforward, cost-efficient manner by directly soliciting stakeholder input on survey items to address specific research questions.8 9 In the context of needs assessment, it can be used to create a priority list of challenges that need to be addressed, along with potential solutions. When the NGT is used, a specific question is presented to the group of participants, then participants individually and silently generate lists of examples in response to the question. Following this, each participant shares each of her or his items one at a time in a round-robin format until all items are shared, which results in a compiled list of items from the entire group. After the comprehensive list is compiled, group discussion occurs among participants and items are removed, reworded or added to the list. Finally, participants vote or rank the items generated in terms of importance or relevance.8 The NGT has been used previously for preliminary item generation and to provide direction for survey development with stakeholder groups that include patients and family members of persons impacted by health conditions,10 including people with type 2 diabetes and multiple sclerosis, and caregivers of people with Parkinson’s disease.8 10–12 In addition to its efficiency and ability to directly incorporate stakeholder input into surveys, the collaborative nature of the NGT may increase stakeholder ownership of research and increase the likelihood that programmes that are developed effectively address stakeholders’ most important needs.13

      Informal caregivers are people who provide support for a family member or friend in need of care due to a health condition.14 There is only limited research on informal caregivers in rare diseases, despite the important role they fill and the emotional and practical challenges they face.15 Many patients with rare diseases have substantial care needs, but there are often few specialised resources available to them through the healthcare system. As such, informal caregivers for patients with rare diseases may undertake a substantial role in caregiving.16–18

      Systemic sclerosis (SSc, scleroderma) is a rare chronic autoimmune disorder, characterised by vascular damage, inflammatory system activation and excessive production of collagen.19 Only one doctoral thesis has considered the experiences of informal caregivers of people with SSc. This unpublished thesis was a qualitative study that included 13 caregivers.20 Consistent with caregiving in other diseases, SSc caregivers who participated in the study reported that their experience as a caregiver involved having to manage additional tasks (eg, household chores), increased negative feelings (eg, guilt) and personal stress, and relationship changes (eg, relational strain with the person with SSc).19 The specific challenges faced by caregivers of persons with SSc, however, have not been studied systematically.

      Developing resources for informal caregivers for persons with SSc may help them manage their role and reduce burden, but information is required on the challenges they face and their preferred support resources. Gathering information from a large number of caregivers is best accomplished via a survey. The objective of this study was to use the NGT in a series of discussions to develop survey items to assess: (1) challenges experienced by informal caregivers of people living with SSc and (2) their preferences for types of support services that could potentially be developed.

      Methods

      Participants and procedures

      We conducted three NGT discussions at the national patient conferences of the Scleroderma Foundation of the USA (two groups) and Scleroderma Canada (one group). Eligible participants were people who had provided unpaid care in the past 12 months to a friend, family member or partner with SSc. It was not required that the caregiver live with the person diagnosed with SSc. Potential participants were emailed study announcements by the Scleroderma Foundation and Scleroderma Canada prior to the conferences. Participants who expressed interest in the study were then contacted by email by the study coordinator who provided them with details about the study, including information about the date and time of NGT discussions, researchers’ credentials, study goals, research questions and the end goal for the project. Prior to each group, paper copies of the consent form were provided to participants, and they were given the opportunity to ask questions about the study.

      Prior to beginning the NGT discussions, participants completed a brief demographic questionnaire that was used to describe participant characteristics. The questionnaire included items about the participants’ age, sex, race/ethnicity, employment status and information about the person for whom they provided care, including SSc diagnosis subtype and years since diagnosis. They also provided caregiving information, including type and length of relationship to the person with SSc, tasks undertaken as part of caregiving and time spent providing care.

      The three NGT discussions ranged in length from 90 to 120 min and were conducted in July 2016 (Scleroderma Foundation) and September 2016 (Scleroderma Canada). The three groups were held in private hotel conference rooms and were moderated by two members of the research team. The first and second groups were moderated by the principal investigator, a male professor trained in clinical psychology and knowledgeable about SSc (BDT) and a female doctoral student in clinical psychology knowledgeable about SSc (DBR). The third group was moderated by DBR and a female doctoral student in counselling psychology with experience in SSc research (STG). A female research assistant was present as an observer in the third group. All moderators had previous experience with discussion-based research. In addition, moderators pilot tested the interview guide among members of a research group from Montreal, Quebec, prior to conducting the caregiver NGT discussions. The pilot test did not result in changes to the protocol, but it resulted in increasing the time allotted for the planned NGT discussions. Participants were informed that the objectives of the NGT discussions were to (1) develop a list of challenges they faced as informal caregivers and (2) develop a list of caregiver support resources that would be helpful to them, as well as the ideal format for delivering these services.

      After the NGT procedures were explained, participants were presented with the first research question: ‘Think about the challenges you have faced since taking on a caregiving role to somebody close to you with SSc’. After being presented with this question, participants were asked to list on a piece of paper the challenges that they have experienced while helping to care for a family member, friend or partner with SSc. Participants developed a list of challenges individually without consultation with other group members. Once completed, participants shared one item at a time from their list in a round-robin format. Each member shared an item in turn until all members had a chance to share an item, then the process began again until every item on each participant’s list had been shared. Participants were instructed not to repeat items that were verbatim to items provided by others but to share any items that seemed to differ, even if only minimally. Participants’ answers were typed on a computer and projected onto a screen as they were provided so that the list could be viewed by the moderators and participants as it developed. Once all items had been shared, group discussion was used to clarify the meaning and wording of items and to remove or merge overlapping items. A consensus process was used, which involved asking participants if they agreed with the wording of items and, for example, if participants thought an item would be better separated into two items. Items were edited based on group feedback until agreement was reached for all decisions.

      In each group, once a list of unique items was agreed on, one of the moderators printed the list of items. Participants then rated the importance of each challenge listed on a scale from 1 to 10, with 1 representing challenges that they did not perceive as personally important to them in their role as an informal caregiver and 10 representing extremely important challenges. After each participant had rated the items, the moderator collected the ratings. Item ratings were collected to inform the removal of items that may have been suggested, but were not especially relevant to caregivers.

      Next, participants were presented with the second research question: ‘Think about services that could be put in place to provide better support to SSc caregivers. What programmes, services or supports would be helpful in your role as a caregiver? How would these programmes, services or other supports operate?’ Participants were asked to write down on a piece of paper any support services that they thought would be helpful and the way in which these services could be delivered. The same process used for answering the first research question was then applied to this research question, and a final master list of support services was developed. As with the first research question, each idea for a support service that was generated was rated independently by each participant on a scale from 1 to 10, with 1 representing support services that they believed they would be unlikely to use and 10 representing services they believed they would very likely use.

      For both research questions, as participants shared their items, if clarification was necessary, probes were used to gain a clearer understanding of the challenges and support services stated (eg, ‘Can you elaborate on that?’ (see online supplementary appendix S1 for interview guide)).

      Supplementary file 1

      Data analysis

      Descriptive analyses were conducted to present demographic data. Many of the challenges and support services items were generated in more than one group. Thus, the research team created a master list that combined all generated items, identified items that overlapped between groups and merged overlapping items. Mean scores were calculated for each unique item. All analyses were conducted with SPSS V.22.0.

      The master list of potential survey items that integrated responses from all three NGT discussions was distributed to members of the research team, including the Scleroderma Caregiver Advisory Team, which comprised eight  informal caregivers of people with SSc. All research team members reviewed the list of items and associated means. The team then made recommendations to remove items that were too vague or repetitive and had the opportunity to suggest new items. An iterative process was used to reword or remove any unclear items, and to incorporate any additional items deemed relevant and important, until consensus on a final list of items was reached.

      To categorise challenges, qualitative content analysis was employed.21 First, relevant literature was reviewed to consider pre-established categories for caregiver challenges. A previous systematic review of 192 articles categorised consequences of caring for people living with cancer into four categories.22 The four categories were used as a starting point to generate categorisations for the challenge items in our study. Challenges were first categorised by two members of the research team who, in collaboration with the principal investigator, developed definitions and rules for each category and refined or added categories as necessary (see online supplementary appendix S2). Another member of the research team, blind to the initial categorisation process, then used the definitions and rules to independently categorise the challenges.

      Results

      Participant and caregiving characteristics

      A total of 13 informal caregivers of persons with SSc (seven women, six men) participated in one of the three NGT discussions (Scleroderma Foundation Group 1=two women; 0 men; Scleroderma Foundation Group 2=three women; two men; Scleroderma Canada Group=two women, four men). None of the participants who came to the group sessions declined to participate or dropped out prior to completing the study.

      Participant sociodemographic and caregiving situation characteristics are presented in table 1. Caregivers ranged in age from 28 to 76 years (mean=59.8 years, SD=12.6). Most caregivers were employed full-time (n=5) or retired (n=7); one caregiver was unemployed. All five caregivers who worked reported that providing care interfered with their job. Participants provided caregiving for a partner (n=8), parent (n=2), friend (n=2), or child (n=1). Caregivers had provided care for an average of 8.9 years (SD=7.8; range = <1 year to 25 years) with an average of 10.2 hours per week of care provided (SD=9.9; range=1–25 hours).

      View this table:
      Table 1

      Sociodemographic characteristics of 13 nominal group technique discussion participants

      NGT discussions: challenge items

      The three groups generated 24, 27 and 38 original caregiving challenge items for a total of 89 (online supplementary appendix S3), although there were duplicate items across groups. Of the 89 original items, 16 items received a mean score for importance between 8.0 and 10.0; 27 items were between 6.0 and 7.9; 39 items were between 4.0 and 5.9; and 7 items were rated <4.0.

      After completion of the groups, the 89 items were distributed to members of the research team and the SSc Caregiver Advisory Team, resulting in the rewording, removal or combining of items. There were 55 unique challenge items remaining after this editing process. Six items were added by the research team and the SSc Caregiver Advisory Committee, resulting in a total of 61 identified challenges (table 2).

      View this table:
      Table 2

      Reduced and categorised list of caregiver generated challenges and item means

      Using a modified set of the categories used in a previous study,22 the 61 challenges were grouped into nine categories (see online supplementary appendix S2 for category definitions). There was 88% agreement of item categorisation between raters. The definitions and rules were used in order to reach a consensus for the placement of each item. As shown in table 2, the nine categories included physical health concerns (n=3 items); financial problems and work or employment problems (n=4 items); role strain (n=8 items); information, resources and support needs (n=15 items); fear, anxiety and uncertainty (n=3 items); general emotional difficulties (n=7 items), emotional difficulties of the care recipient (n=8 items); changes in relationship dynamics with care recipient (n=10 items); and changes in social interactions (n=3 items).

      NGT discussions: support service items

      The three groups generated 13, 13 and 15 original items reflecting preferred support service options (see online supplementary appendix S4). Among the 41 items, 12 received a score reflecting likelihood of using the service between 9.0 and 10.0; 6 between 8.0 and 8.9; 15 between 6.0 and 7.9; and 8 less than 6.0. There was a high degree of duplication of items across groups. Thus, the 41 total items were reduced by the research team to 18 unique items, and these items were reviewed and edited by the research team, as necessary (table 3). These 18 items included support services delivered online (n=7), by telephone (n=2), hard-copy resources (n=2) and in-person delivery (n=7).

      View this table:
      Table 3

      Reduced list of caregiver-generated support services and item means

      Discussion

      In the present study, we used a novel NGT method to identify challenges faced by informal caregivers of persons with SSc and potential support services to address these challenges. Based on the quantity and variety of items that caregivers generated, NGT discussions were effective for developing items for our planned needs assessment survey of SSc caregivers. Use of the NGT enabled caregivers to directly share their perspectives and provided a mechanism for direct caregiver input in the development of survey that will be distributed to a larger sample of caregivers to persons with SSc.

      A final list of 61 survey items that reflect unique challenges related to caregiving in SSc was generated. These challenges included physical health concerns; financial problems and work or employment problems; role strain; information, resources and support needs; fear, anxiety and uncertainty; general emotional difficulties; emotional difficulties of the care recipient; changes in relationship dynamics with the care recipient; and changes in social interactions. The largest number of items reflected challenges related to unmet information, resource and support needs. Caregivers also generated items that reflected 18 support services that could be delivered to help address difficult aspects of caregiving. Support services that were delivered online, by telephone, in-person and through hard-copy resources were all identified by caregivers as being potentially helpful. Online or in-person delivery of support services that focused on providing education and emotional support were the most common suggestions.

      To our knowledge, this is the first study to gather information on challenges from informal caregivers of people living with SSc using NGT discussions. Our findings, however, can be compared with a thesis that included individual interviews with 13 caregivers of persons with SSc.20 Several of the challenges generated during the NGT discussions that related to general emotional difficulties are similar to themes identified in the thesis, including guilt, frustration and stress. Further, financial strain and career adjustments, relational strain and difficulty managing multiple roles were reported as a challenges of caregiving in our NGT discussions and in previously reported interviews.20

      Our results can also be compared with studies of caregivers to persons with more common diseases, including Huntington’s disease, cancer and Alzheimer’s disease. Caregivers to persons with more common diseases have described several similar challenges, including understanding and managing the patient’s medical needs, changes to the patient–caregiver relationship, managing disease-related problems and concerns about the care recipient not receiving adequate care.22–25 Our findings should also be considered in the context of rare diseases. Previous literature has suggested that rare diseases pose additional challenges for both the caregiver and the person living with the disease.17 26 27 Our results are in line with this research as the most commonly reported challenges related to the rarity of the disease and difficulty obtaining necessary information, resources and support.

      There are currently no formal resources in place for SSc caregivers, and no previous research has administered support service interventions among caregivers to persons with SSc. There are, however, many support services that have been developed to lessen the burden associated with caregiving in common diseases,28–31 and some of these may be able to be adapted for SSc caregivers. For example, caregiver support groups, teleconference-based interventions and educational sessions32–36 have been provided for caregivers of persons with dementia, asthma, allogeneic haematopoietic stem cell transplantation, and cancer and were identified in our study as support services that SSc caregivers would be likely to use.

      Strengths and limitations

      The present study has important strengths. For example, no other studies have used NGT discussions as a way of eliciting information from caregivers of people with rare diseases. Applying this methodology allowed us to generate a robust list of potential survey items in the words of stakeholders themselves and provided initial quantitative ratings of the importance of challenges and likelihood of service use for the items generated. Further, our research team included our Scleroderma Caregiver Advisory Team, which comprised caregivers to persons with SSc. This allowed for the incorporation of stakeholder input at each stage of the research and will aid in the facilitation of follow-up survey work and the eventual implementation of the findings. An advantage of the NGT compared with standard focus groups is that focus groups use open-ended questions to solicit discussion, and researchers must then develop survey items from the themes that were discussed. The NGT allowed for direct caregiver input that did not require researchers to interpret and extrapolate from what participants shared in the discussion to generate survey items.

      There are limitations to consider when interpreting the results of this study. First, the NGT discussions were held at national patient conferences. This may have influenced the characteristics of participants and the generalisability of results as only caregivers attending the 2016 American or Canadian conferences were eligible for participation in the groups. Specifically, our sample may have overincluded caregivers who are well connected to the SSc community. Second, participants only rated items one time in the groups. NGT discussions sometimes ask participants to re-score items after discussing the results of the first round of scoring, similar to a Delphi process.37 Participants were only asked to rate items once and ratings were completed independently rather than discussed with other participants. Previous studies, however, have found that when asked participants state that item ratings would generally remain unchanged if asked to rate items a second time.38 Third, participation in the NGT discussions was restricted to caregivers who were literate. Lastly, four challenges were removed from our list of unique items due to the vague content of these items. During the NGT discussions, gathering additional specifications about the challenges may have been helpful to ensure that the items elicited from participants were retained. Despite these limitations, the study provides important information that can be used to inform the development of a survey to determine caregivers’ most important challenges, and the support services that caregivers would be most likely to use.

      Conclusion

      The NGT was an efficient approach for gathering caregiver input to aid in the development of survey items. We found that caregivers of persons with SSc face many challenges and have substantial unmet needs. Some of the most important challenges identified were related to information, resources and support needs. A range of possible support services were identified, with caregivers reporting being most likely to use services delivered in-person or through online platforms for education and emotional support, including internet-based psychological and emotional self-help tools. The findings from the present study suggest that programmes offered online may result in a greater likelihood of caregiver participation. The results of the present study will be used to construct a survey that will be disseminated online to a larger number of SSc caregivers in order to better understand the relative importance of the challenges identified and the likelihood of use of possible support services. The results of the present study, combined with survey results, will be used to develop SSc caregiver support services.

      References

      1. 1.↵
        2. O’Haire C ,
        3. McPheeters M ,
        4. Nakamoto E , et al
        . Engaging stakeholders to identify and prioritize future research needs. Rockville (MD): Agency for Healthcare Research and Quality (US), 2011. Report No: 11-EHC044-EF.
      2. 2.↵
        2. Romine W ,
        3. Sadler TD ,
        4. Presley M , et al
        . Student Interest in Technology and Science (SITS) survey: development, validation, and use of a new instrument. Int J Sci Math Educ 2014;12:261–83.doi:10.1007/s10763-013-9410-3
        OpenUrlCrossRef
      3. 3.↵
        2. Delisle VC ,
        3. Gumuchian ST ,
        4. Pelaez S , et al
        . Reasons for non-participation in scleroderma support groups. Clin Exp Rheumatol 2016;34(Suppl 100):56–62.
        OpenUrl
      4. 4.↵
        2. Thomson R ,
        3. Lewalle P ,
        4. Sherman H , et al
        . Towards an international classification for patient safety: a delphi survey. Int J Qual Health Care 2009;21:9–17.doi:10.1093/intqhc/mzn055
        OpenUrlCrossRefPubMedWeb of Science
      5. 5.↵
        2. Francis J ,
        3. Eccles MP ,
        4. Johnston M , et al
        . Constructing questionnaires based on the theory of planned behaviour: A manual for health services researchers. Newcastle upon Tyne, UK: Centre for Health Services Research, University of Newcastle upon Tyne, 2004.
      6. 6.↵
        2. Asadi-Lari M ,
        3. Tamburini M ,
        4. Gray D
        . Patients' needs, satisfaction, and health related quality of life: towards a comprehensive model. Health Qual Life Outcomes 2004;2:32.doi:10.1186/1477-7525-2-32
        OpenUrlCrossRefPubMed
      7. 7.↵
        2. Smithson J
        . Using and analysing focus groups: Limitations and possibilities. Int J Soc Res Methodol 2000;3:103–19.doi:10.1080/136455700405172
        OpenUrl
      8. 8.↵
        2. Harvey N ,
        3. Holmes CA
        . Nominal group technique: an effective method for obtaining group consensus. Int J Nurs Pract 2012;18:188–94.doi:10.1111/j.1440-172X.2012.02017.x
        OpenUrlCrossRefPubMed
      9. 9.↵
        2. Delbecq AL ,
        3. van de Ven AH ,
        4. Gustafson DH
        . Group techniques for program planning, a guide to nominal group and Delphi processes. Glenview, IL: Scott, Foresman and Company, 1975.
      10. 10.↵
        2. Chasens ER ,
        3. Olshansky E
        . Daytime sleepiness, diabetes, and psychological well-being. Issues Ment Health Nurs 2008;29:1134–50.doi:10.1080/01612840802319878
        OpenUrlCrossRefPubMed
      11. 11.↵
        2. Kleiner-Fisman G ,
        3. Martine R ,
        4. Lang AE , et al
        . Development of a non-motor fluctuation assessment instrument for Parkinson disease. Parkinsons Dis 2011;2011:1–13.doi:10.4061/2011/292719
        OpenUrlCrossRef
      12. 12.↵
        2. Kremer IE ,
        3. Evers SM ,
        4. Jongen PJ , et al
        . Identification and prioritization of important attributes of disease-modifying drugs in decision making among patients with multiple sclerosis: a nominal group technique and best-worst scaling. PLoS One 2016;11:e0164862.doi:10.1371/journal.pone.0164862
      13. 13.↵
        2. Vella K ,
        3. Goldfrad C ,
        4. Rowan K , et al
        . Use of consensus development to establish national research priorities in critical care. BMJ 2000;320:976–80.doi:10.1136/bmj.320.7240.976
        OpenUrlAbstract/FREE Full Text
      14. 14.↵
        2. Reinhard SC ,
        3. Given B ,
        4. Petlick NH , et al
        . Supporting family caregivers in providing care. In: Hughes RG , ed. Patient safety and quality: an evidence-based handbook for nurses. Rockville, MD: Agency for Healthcare Research and Quality (US), 2008:341–63.
      15. 15.↵
        2. Adelman RD ,
        3. Tmanova LL ,
        4. Delgado D , et al
        . Caregiver burden: a clinical review. JAMA 2014;311:1052–60.doi:10.1001/jama.2014.304
        OpenUrlCrossRefPubMedWeb of Science
      16. 16.↵
        2. Rode J
        . Rare diseases: understanding this public health priority. 2005 http://www.eurordis.org/IMG/pdf/princeps_document-EN.pdf (accessed 8 Jan 2017).
      17. 17.↵
        2. Kole A ,
        3. Faurisson F
        . The Voice of 12,000 patients: experiences and expectations of rare disease patients on diagnosis and care in Europe, 2009. https://www.eurordis.org/IMG/pdf/voice_12000_patients/EURORDISCARE_FULLBOOKr.pdf. (accessed 8 Jan 2017).
      18. 18.↵
        Wold Health Organization. World health organization, Department of essential drugs and medicines policy. Bundesgesundheitsbla 2005;48:221–31.
        OpenUrl
      19. 19.↵
        2. Bernatsky S ,
        3. Joseph L ,
        4. Pineau CA , et al
        . Scleroderma prevalence: demographic variations in a population-based sample. Arthritis & Rheumatism 2009;61:400–4.doi:10.1002/art.24339
        OpenUrlCrossRefPubMedWeb of Science
      20. 20.↵
        2. Maril V
        . Caring for a loved one with scleroderma: experiences of caregivers. Texas, USA : Texas Woman’s University, 2012.
      21. 21.↵
        2. Mayring P
        . Qualitative Content Analysis. Forum: qualitative social research. http://nbn-resolving.de/urn:nbn:de:0114-fqs0002204. (accessed 8 Jan 2017).
      22. 22.↵
        2. Stenberg U ,
        3. Ruland CM ,
        4. Miaskowski C
        . Review of the literature on the effects of caring for a patient with cancer. Psychooncology 2010;19:1013–25.doi:10.1002/pon.1670
        OpenUrlCrossRefPubMed
      23. 23.↵
        2. Coristine M ,
        3. Crooks D ,
        4. Grunfeld E , et al
        . Caregiving for women with advanced breast cancer. Psychooncology 2003;12:709–19.doi:10.1002/pon.696
        OpenUrlCrossRefPubMed
      24. 24.↵
        2. Mahoney DF
        . A content analysis of an Alzheimer family caregivers virtual focus group. Am J Alzheimers Dis 1998;13:309–16.doi:10.1177/153331759801300606
        OpenUrl
      25. 25.↵
        2. Williams JK ,
        3. Barnette JJ ,
        4. Reed D , et al
        . Development of the Huntington disease family concerns and strategies survey from focus group data. J Nurs Meas 2010;18:83–99.doi:10.1891/1061-3749.18.2.83
        OpenUrlPubMed
      26. 26.↵
        2. Doyle M
        . Peer support and mentorship in a us rare disease community: findings from the cystinosis in emerging adulthood study. Patient 2015;8:65–73.doi:10.1007/s40271-014-0085-9
        OpenUrl
      27. 27.↵
        2. Gowran RJ ,
        3. Kennan A ,
        4. Marshall S , et al
        . Adopting a sustainable community of practice model when developing a service to support patients with Epidermolysis Bullosa (EB): a stakeholder-centered approach. Patient 2015;8:51–63.doi:10.1007/s40271-014-0097-5
        OpenUrl
      28. 28.↵
        2. Pinquart M ,
        3. Sörensen S
        . Helping caregivers of persons with dementia: which interventions work and how large are their effects? Int Psychogeriatr 2006;18:577–95.doi:10.1017/S1041610206003462
        OpenUrlCrossRefPubMedWeb of Science
      29. 29.↵
        2. Thompson CA ,
        3. Spilsbury K ,
        4. Hall J , et al
        . Systematic review of information and support interventions for caregivers of people with dementia. BMC Geriatr 2007;7:18.doi:10.1186/1471-2318-7-18
        OpenUrlCrossRefPubMed
      30. 30.↵
        2. Boots LM ,
        3. de Vugt ME ,
        4. van Knippenberg RJ , et al
        . A systematic review of Internet-based supportive interventions for caregivers of patients with dementia. Int J Geriatr Psychiatry 2014;29:331–44.doi:10.1002/gps.4016
        OpenUrl
      31. 31.↵
        2. Selwood A ,
        3. Johnston K ,
        4. Katona C , et al
        . Systematic review of the effect of psychological interventions on family caregivers of people with dementia. J Affect Disord 2007;101:75–89.doi:10.1016/j.jad.2006.10.025
        OpenUrlCrossRefPubMedWeb of Science
      32. 32.↵
        2. Chien LY ,
        3. Chu H ,
        4. Guo JL , et al
        . Caregiver support groups in patients with dementia: a meta-analysis. Int J Geriatr Psychiatry 2011;26:1089–98.doi:10.1002/gps.2660
        OpenUrlCrossRefPubMed
      33. 33.↵
        2. Neuharth-Pritchett S ,
        3. Getch YQ
        . The effectiveness of a brief asthma education intervention for child care providers and primary school teachers. Early Child Educ J 2016;44:555–61.doi:10.1007/s10643-015-0751-0
        OpenUrl
      34. 34.↵
        2. Bevans M ,
        3. Wehrlen L ,
        4. Castro K , et al
        . A problem-solving education intervention in caregivers and patients during allogeneic hematopoietic stem cell transplantation. J Health Psychol 2014;19:602–17.doi:10.1177/1359105313475902
        OpenUrlCrossRefPubMed
      35. 35.↵
        2. Bormann J ,
        3. Warren KA ,
        4. Regalbuto L , et al
        . A spiritually based caregiver intervention with telephone delivery for family caregivers of veterans with dementia. Fam Community Health 2009;32:345–53.doi:10.1097/FCH.0b013e3181b91fd6
        OpenUrlPubMed
      36. 36.↵
        2. Walsh SM ,
        3. Schmidt LA
        . Telephone support for caregivers of patients with cancer. Cancer Nurs 2003;26:448???453–53.doi:10.1097/00002820-200312000-00004
        OpenUrl
      37. 37.↵
        2. Hsu C-C ,
        3. Sandford BA
        . The Delphi technique: making sense of consensus. PARE 2007;12:1–8.
        OpenUrl
      38. 38.↵
        2. Ng V ,
        3. Sargeant JM
        . A stakeholder-informed approach to the identification of criteria for the prioritization of zoonoses in Canada. PLoS One 2012;7:e29752.doi:10.1371/journal.pone.0029752

      Footnotes

      • Contributors DBR and BDT were responsible for the study conception. DBR, STG and BDT contributed to data collection. DBR, MCA, KAT, STG, VLM, MH, BDT and the SCAT contributed to data analysis and interpretation. DBR, MCA and BDT contributed to drafting the manuscript. All authors provided a critical revision of the manuscript and approved the final version of the manuscript. BDT is the guarantor. The authors had full access to the data and take responsibility for the integrity of the data and the accuracy of the data analysis.

      • Funding This research was supported by a grant from the Rare Disease Foundation and the BC Children’s Hospital Foundation (Funding Reference Number 173-16; PI DBR).

      • Competing interests DBR was supported by a Vanier Graduate Scholarship. STG was supported by a Fonds de Recherché Santé Québec Doctoral Award. BDT was supported by a Fonds de recherche du Québec – Santé researcher salary award.

      • Patient consent Obtained.

      • Ethics approval The Research Ethics Committee of the Jewish General Hospital, Montreal, Quebec, Canada.

      • Provenance and peer review Not commissioned; externally peer reviewed.

      • Data sharing statement No additional data are available.

      • Collaborators Scleroderma Caregiver Advisory Team Members: Marcia Greiten, Goleta, California, USA; Christy McCaffrey, Philadelphia, Pennsylvania, USA; Judi McDonald, Bedford, Nova Scotia,Canada; John Michalski, Macomb, Michigan, USA; Mathieu Ross, Quebec City, Quebec, Canada; Lisa Spinney, Berlin, Massachusetts, USA; Terry Stacey, Oshawa, Ontario, Canada;Linda Tarantino, Larkspur, California, USA.