INTRODUCTION
Conducting pragmatic trials testing complex interventions in general practice is important for determining which interventions work in the real-world. However, pragmatic trials present many methodological and logistical challenges. In this article, we share our experience and lessons from the SPACE trial (Safer Prescribing And Care for the Elderly) in the hope that others might avoid some of the pitfalls (Box 1).1 SPACE was conducted in New Zealand general practice where there is no established infrastructure supporting practice-based research.
| Challenges | Potential solutions |
|---|
| Noise in the real-world |
Robust intervention, with strong theoretical underpinnings, capable of impact despite contextual changes Established practice network that rigorously tests quality improvement initiatives via randomised controlled trials
|
| Difficulty recruiting practices |
Topic that is interesting and important to GPs Participation remunerated and not too onerous Established practice network that supports long-term engagement
|
| Between group differences |
|
| Poor intervention uptake |
Piloting to determine feasibility, acceptability, and utility Support practice engagement
|
| Loss to follow up |
|
| Randomisation error |
|
Box 1. Lessons from SPACE:1 challenges conducting pragmatic trials in general practice and potential solutions
NOISE IN THE REAL-WORLD
SPACE was a pragmatic trial testing an intervention to support safer prescribing. Unfortunately, soon after securing funding for the trial, a non-trial quality improvement (QI) initiative was introduced in the same region, targeting the same prescribing topic. This reduced the pool of practices for recruitment (we excluded participating practices), introduced confounding by increasing awareness of the prescribing issue, and since some trial practices joined the initiative during follow up, contaminated our results. In traditional explanatory trials that test whether an intervention can work under ideal conditions, it would be possible to exclude practices to minimise confounding.2 However, in pragmatic trials seeking to test effectiveness in the real-world, data from all enrolled practices must be included.3–5 Contamination and dilution of intervention effect are a feature of the real-world. Any intervention needs to be sufficiently robust to produce an effect regardless. Another possible solution is to have an established network of practices that integrates QI initiatives and research, for example, using rigorous controlled designs such as cross-over designs, with half of practices randomised to intervention A and the other half to intervention B then cross-over.
DIFFICULTY RECRUITING PRACTICES
Recruitment is one of the most challenging aspects of practice-based research. GPs are busy people and participation in research is often not a priority. For SPACE, only 36% of eligible practices agreed to participate.1 The most commonly cited reason for non-participation was ‘lack of time’. The burden of research may mean that only certain types of practices participate, introducing bias. Low participation rates also limit generalisability of results. For pragmatic trials, the topic must be sufficiently interesting and important to GPs, the intervention not too demanding, and participation adequately remunerated. Structural support for practice-based research, such as an established network, would also help to create opportunities for practices and GPs to have long-term engagement with research including identifying questions, and piloting and testing interventions.
BETWEEN GROUP DIFFERENCES
For SPACE, we chose a cluster randomised controlled trial design with practices as the unit of randomisation, stratified by size (small, medium, and large) and location (region A or B).6 Randomising by cluster allows an intervention to be delivered at practice- and GP-level to minimise contamination of intervention.7 However, clustering risks recruitment bias and imbalance of participant characteristics through variation between practices. Unfortunately, in SPACE, the rate of high-risk prescribing at baseline was substantially higher in intervention (7.1%) compared with control practices (6.4%) suggesting between group differences that could affect response to the intervention. Accurate estimates of intraclass coefficients to determine appropriate sample size inflation factors, careful stratification, and using software to randomly allocate practices can help to minimise differences between groups. An alternative option, if contamination is likely to be <30%, is to conduct an individual randomised controlled trial and inflate the sample size to allow for dilution of effect.7
POOR INTERVENTION UPTAKE
Poor intervention uptake can also be a factor. Traditional explanatory trials can use a ‘run-in’ period to ensure that only engaged practices are enrolled in the study and exclude as ‘protocol violations’ any practices that do not engage. However, since the goal of pragmatic trials is to test effectiveness including intervention uptake, data from all practices must be included.8 In SPACE, only 70% of GPs in intervention practices engaged.1 Piloting the intervention to determine feasibility, acceptability, and utility is important, and supporting participation by keeping in close contact with practices and building relationships can also help.9
LOSS TO FOLLOW UP
A pragmatic trial has clinically important outcomes and uses data collection methods that are minimally disruptive to practices. In SPACE, we used pre-defined automated extraction of clinical data via practice management software, which was convenient for practices. Unfortunately, during the course of the trial two large intervention practices changed their practice management systems resulting in substantial loss-to-follow-up (14.7% intervention participants).1 Trials must be sufficiently powered to accommodate predicted loss of sample and still achieve results that are significant.
RANDOMISATION ERROR
Randomisation and blinding are important to avoid bias and ensure that any effect can be attributed to the intervention.10 Double-blinding is not possible in pragmatic trials as GPs know they are receiving the intervention. However, concealment of allocation from analysts remains important: lack of concealment has been found to be associated with a 30% exaggeration of treatment effect.8 For SPACE, we engaged a (blinded) biostatistician to generate a random sequence assignment (‘I’ for intervention or ‘C’ for control) for each block of de-identified practices. The biostatistician emailed the assignment to the principal investigator who transcribed the assignment into an excel spreadsheet, allocating practices to their assigned group. Unfortunately, for one block of six practices the principal investigator made a transcription error. Struck by the beauty of a particular random sequence, she inadvertently inverted the random group assignment entering into the spreadsheet ‘ICICIC’ instead of ‘CICICI’, thereby allocating practices to the inverse of their random sequence assignment. This error was not realised until completion of the trial at the time of analysis and was reported in the trial write-up.1 It is important to report recruitment, randomisation, and treatment errors, and any variation from protocol, to maintain trial integrity and enable the reader to decide how errors may bias results, although underreporting of trial errors remains common.11–13 Most modern trials avoid the risk of human error by using automated software systems to randomly allocate practices.
RANDOMISATION AND ALLOCATION
The transcription error raised the question of whether the primary intention to treat analysis should be by initial ‘randomisation’ or by actual ‘allocation’. It is well accepted that the primary analysis should be by ‘randomisation’, but usually ‘randomisation’ and ‘allocation’ are one and the same process. However, for SPACE, randomisation and allocation were distinct processes, with ‘randomisation’ comprising the random sequence assignment and ‘allocation’ comprising transcription of the assignment. Analysis by ‘randomisation’, or random sequence assignment, risked a type 2 error (false negative) introducing bias towards the null hypothesis.8 For example, if there had been inadvertent inversion of every random sequence and the intervention had a true effect, then the conclusion would be, not that the intervention supported safer prescribing, but that it made prescribing worse. On the other hand, analysis by ‘allocation’ risked an overestimate of effect.8 Since the principal investigator was not blinded (having recruited the practices), she could have been biased in her error. Neither analysis was methodologically pure (there had been a protocol violation). However, if the transcription error had been random, then arguably the allocation was also random albeit the inverse of the original random assignment. After seeking advice, we settled on primary analysis by allocation and presented the analysis by randomisation as a secondary analysis.
There are a number of other recognised challenges that were successfully navigated in the SPACE trial including ethical approval and contractual arrangements. Despite the many difficulties encountered (see Box 1), SPACE was completed and has added to the evidence base of what works in practice. An established practice-based research network, where researchers and practices work together to develop robust evidence, could overcome many of the challenges, facilitating practice-based research to improve primary health care and outcomes for patients.
Acknowledgments
Auckland Medical Research Foundation, New Zealand for funding the SPACE trial; participating general practices and GPs; and Professors Paul Glasziou and Nicholas Zwar for helpful suggestions.
Notes
Provenance
Freely submitted; externally peer reviewed.
Competing interests
The authors have declared no competing interests.
- © British Journal of General Practice 2022
