A full systematic review was completed in 2 weeks using automation tools: a case study

doi:10.1016/j.jclinepi.2020.01.008

Journal of Clinical Epidemiology

Volume 121, May 2020, Pages 81-90

https://doi.org/10.1016/j.jclinepi.2020.01.008 Get rights and content

Abstract

Background and Objectives

Systematic reviews (SRs) are time and resource intensive, requiring approximately 1 year from protocol registration to submission for publication. Our aim was to describe the process, facilitators, and barriers to completing the first 2-week full SR.

Study Design and Setting

We systematically reviewed evidence of the impact of increased fluid intake, on urinary tract infection (UTI) recurrence, in individuals at risk for UTIs. The review was conducted by experienced systematic reviewers with complementary skills (two researcher clinicians, an information specialist, and an epidemiologist), using Systematic Review Automation tools, and blocked off time for the duration of the project. The outcomes were time to complete the SR, time to complete individual SR tasks, facilitators and barriers to progress, and peer reviewer feedback on the SR manuscript. Times to completion were analyzed quantitatively (minutes and calendar days); facilitators and barriers were mapped onto the Theoretical Domains Framework; and peer reviewer feedback was analyzed quantitatively and narratively.

Results

The SR was completed in 61 person-hours (9 workdays; 12 calendar days); accepted version of the manuscript required 71 person-hours. Individual SR tasks ranged from 16 person-minutes (deduplication of search results) to 461 person-minutes (data extraction). The least time-consuming SR tasks were obtaining full-texts, searches, citation analysis, data synthesis, and deduplication. The most time-consuming tasks were data extraction, write-up, abstract screening, full-text screening, and risk of bias. Facilitators and barriers mapped onto the following domains: knowledge; skills; memory, attention, and decision process; environmental context and resources; and technology and infrastructure. Two sets of peer reviewer feedback were received on the manuscript: the first included 34 comments requesting changes, 17 changes were made, requiring 173 person-minutes; the second requested 13 changes, and eight were made, requiring 121 person-minutes.

Conclusion

A small and experienced systematic reviewer team using Systematic Review Automation tools who have protected time to focus solely on the SR can complete a moderately sized SR in 2 weeks.

Section snippets

Background

Systematic reviews (SRs) synthesize evidence to answer a specific question, using methods that are transparent and reproducible. They are considered the highest-level of evidence to underpin clinical and policy decisions.

However, SRs are time and resource intensive, requiring a median of five researchers and 41 weeks to submit to a journal [personal communication, Kathryn Kaiser; [1]]. A median-sized SR search yields 1,781 references (range: 27–92,020) and requires a title/abstract screen of

The systematic review

We completed an SR of randomized controlled trials (RCTs) assessing the impact of increased fluid intake on urinary tract infection (UTI) recurrence, antimicrobial use, and UTI symptoms, in individuals at risk for UTIs [5]. We included RCTs which compared interventions involving increased fluid intake (e.g., water, juice) to those not involving increased fluids. Searches identified 1,694 references; eight trials were included, and 4 meta-analyses were conducted (Table 1). This was a full SR,

Time to complete the full systematic review

The SR was completed in 12 calendar days (between 21 January and 1 February 2019), working across a 5-day work week and a 4-day work week (because of a public holiday) (Table 4). A standard work week at our university is 37.5 hours (7.5 hours a day).

The SR commenced on 21 January 2019, with an all-author meeting (A.M.S., J.C., C.D.M., and P.G.) during which two authors (P.G. and C.D.M.) disclosed the PICO question to the rest of the team (A.M.S. and J.C.). The work on the protocol commenced

Discussion

Our SR team included four members (compared to a median of five for an SR), required screening of 1,694 studies (slightly less than the median of 1,781), and was completed in 61 person-hours or 9 working days. The time to journal submission was 66 person-hours (16 calendar days), which represents a considerable improvement on the median time to journal submission of 41 weeks (Borah et al. 2017). The final, publishable version of the manuscript required 71 person-hours.

A recent study, evaluating

Conclusion

A small and experienced SR team, using SRA tools who have protected time to focus solely on the SR can complete a moderately sized SR in 2 weeks.

Acknowledgments

The authors would like to thank Kathryn Kaiser for providing additional data from the Borah et al. 2017 study and Ba’ Pham for feedback on the manuscript.

References (14)

B. Pham et al.
Improving the conduct of systematic reviews: a process mining perspective
J Clin Epidemiol
(2018)
R. Borah et al.
Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry
BMJ Open
(2017)
E. Beller et al.
Making progress with the automation of systematic reviews: principles of the international collaboration for the automation of systematic reviews (ICASR)
Syst Rev
(2018)
A.M. O'Connor et al.
Moving toward the automation of the systematic review process: a summary of discussions at the second meeting of International Collaboration for the Automation of Systematic Reviews (ICASR)
Syst Rev
(2018)
A.J. van Altena et al.
Usage of automation tools in systematic reviews
Res Synth Methods
(2019)
A. Scott et al.
Increased fluid intake to prevent urinary tract infections: systematic review and meta-analysis
Br J Gen Pract
(2020)

There are more references available in the full text version of this article.

Cited by (206)

Facilitating public involvement in research about healthcare AI: A scoping review of empirical methods
2024, International Journal of Medical Informatics
With the recent increase in research into public views on healthcare artificial intelligence (HCAI), the objective of this review is to examine the methods of empirical studies on public views on HCAI. We map how studies provided participants with information about HCAI, and we examine the extent to which studies framed publics as active contributors to HCAI governance.
We searched 5 academic databases and Google Advanced for empirical studies investigating public views on HCAI. We extracted information including study aims, research instruments, and recommendations.
Sixty-two studies were included. Most were quantitative (N = 42). Most (N = 47) reported providing participants with background information about HCAI. Despite this, studies often reported participants’ lack of prior knowledge about HCAI as a limitation. Over three quarters (N = 48) of the studies made recommendations that envisaged public views being used to guide governance of AI.
Provision of background information is an important component of facilitating research with publics on HCAI. The high proportion of studies reporting participants’ lack of knowledge about HCAI as a limitation reflects the need for more guidance on how information should be presented. A minority of studies adopted technocratic positions that construed publics as passive beneficiaries of AI, rather than as active stakeholders in HCAI design and implementation.
This review draws attention to how public roles in HCAI governance are constructed in empirical studies. To facilitate active participation, we recommend that research with publics on HCAI consider methodological designs that expose participants to diverse information sources.
Neurodiversity and cognitive load in online learning: A systematic review with narrative synthesis
2024, Educational Research Review
This systematic review with narrative synthesis aimed to examine the available evidence on the relationship between neurodiversity and cognitive load in online learning. Despite the known relationship between working memory impairments and neurodiversity, there has been a lack of systematic investigation into how cognitive load impacts neurodivergent students in online learning environments. This review, which includes 90 studies conducted in 21 countries, reveals that the majority (92%) did not consider neurodiversity as a potential factor influencing cognitive load in online learning. The few that did suggest distinct patterns of cognitive load in online learning for neurodivergent students, suggesting a need for more research in this area. The results also point to the potential for methodological advancements in measuring cognitive load in online learning, highlighting that 80% of studies with only neurotypical participants and 60% of studies including neurodivergent participants relied solely on subjective measures of cognitive load. This review contributes to the field by offering a comprehensive overview of our current knowledge of the relationship between neurodiversity and cognitive load in online learning, identifying significant gaps and suggesting directions for future studies.
Stakeholder perspectives towards diagnostic artificial intelligence: a co-produced qualitative evidence synthesis
2024, eClinicalMedicine
Diagnosis is a cornerstone of medical practice. Worldwide, there is increased demand for diagnostic services, exacerbating workforce shortages. Artificial intelligence (AI) technologies may improve diagnostic efficiency, accuracy, and access. Understanding stakeholder perspectives is key to informing implementation of complex interventions. We systematically reviewed the literature on stakeholder perspectives on diagnostic AI, including all English-language peer-reviewed primary qualitative or mixed-methods research.
We searched PubMed, Ovid MEDLINE/Embase, Scopus, CINAHL and Web of Science (22/2/2023 and updated 8/2/2024). The Critical Appraisal Skills Programme Checklist informed critical appraisal. We used a ‘best-fit’ framework approach for analysis, using the Non-adoption, Abandonment, Scale-up, Spread, Sustainability (NASSS) framework. This study was pre-registered (PROSPERO CRD42022313782).
We screened 16,577 articles and included 44. 689 participants were interviewed, and 402 participated in focus groups. Four stakeholder groups were described: patients, clinicians, researchers and healthcare leaders. We found an under-representation of patients, researchers and leaders across articles. We summarise the differences and relationships between each group in a conceptual model, hinging on the establishment of trust, engagement and collaboration. We present a modification of the NASSS framework, tailored to diagnostic AI.
We provide guidance for future research and implementation of diagnostic AI, highlighting the importance of representing all stakeholder groups. We suggest that implementation strategies consider how any proposed software fits within the extended NASSS-AI framework, and how stakeholder priorities and concerns have been addressed.
RK is supported by an NIHR Doctoral Research Fellowship grant (NIHR302562), which funded patient and public involvement activities, and access to Covidence.
Cognitive ability in early life and risk of depression in adulthood: A systematic review and meta-analysis
2024, Journal of Affective Disorders
There is an established association between cognitive ability and risk of depression, though the direction of this association is unclear. Measuring cognitive ability in childhood, prior to the diagnosis of depression, could help to understand whether childhood cognitive ability is associated with a later diagnosis of depression. This systematic review and meta-analysis explored the association between childhood cognitive ability and risk of depression in adulthood.
We searched five databases to January 2024. We included studies that assessed cognitive ability in childhood (<18 years) and depression in adulthood. We excluded studies with very specific populations. We pooled each study's most-adjusted correlation coefficient in a random-effects meta-analysis. When studies reported a dichotomous outcome (depression/no depression), we converted the effect size to a correlation coefficient. Subgroup analyses were performed to explore sources of heterogeneity.
18 articles (19 cohorts) were included. There was no association between childhood cognitive ability and depression in adulthood (20 sample populations, N = 45,786, r = −0.04, 95 % CI = -0.09 to 0.01, p = 0.09). Neither age at cognitive assessment, length of follow-up, using a continuous/categorical measure of depression, or sex, significantly influenced the association. We rated most studies as having moderate risk of bias.
We limited the literature search to studies written in English. Existing studies were also heterogeneous, often adjusting for a variety of covariates.
Our meta-analysis found no association between childhood cognitive ability and depression in adulthood. Future, longitudinal population-level studies should endeavour to control for potential mediators across the life-course (e.g., demographic and environmental factors).
Digital health interventions for postoperative recovery in children: a systematic review
2024, British Journal of Anaesthesia
Digital health interventions offer a promising approach for monitoring during postoperative recovery. However, the effectiveness of these interventions remains poorly understood, particularly in children. The objective of this study was to assess the efficacy of digital health interventions for postoperative recovery in children.
A systematic review was conducted following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, with the use of automation tools for searching and screening. We searched five electronic databases for randomised controlled trials or non-randomised studies of interventions that utilised digital health interventions to monitor postoperative recovery in children. The study quality was assessed using Cochrane Collaboration's Risk of Bias tools. The systematic review protocol was prospectively registered with PROSPERO (CRD42022351492).
The review included 16 studies involving 2728 participants from six countries. Tonsillectomy was the most common surgery and smartphone apps (WeChat) were the most commonly used digital health interventions. Digital health interventions resulted in significant improvements in parental knowledge about the child's condition and satisfaction regarding perioperative instructions (standard mean difference=2.16, 95% confidence interval 1.45–2.87; z=5.98, P<0.001; I²=88%). However, there was no significant effect on children's pain intensity (standard mean difference=0.09, 95% confidence interval −0.95 to 1.12; z=0.16, P=0.87; I²=98%).
Digital health interventions hold promise for improving parental postoperative knowledge and satisfaction. However, more research is needed for child-centric interventions with validated outcome measures. Future work should focus development and testing of user-friendly digital apps and wearables to ease the healthcare burden and improve outcomes for children.
PROSPERO (CRD42022351492).
Common barriers and enablers to the use of non-drug interventions for managing common chronic conditions in primary care: an overview of reviews
2024, BMC Primary Care

View all citing articles on Scopus

: Funding: The systematic review described in the manuscript was conducted as part of the work of the Centre of Research Excellence in Minimising Antibiotic Resistance in the Community (CRE-MARC), funded by the National Health and Medical Research Council (NHMRC), Australia. The funder had no involvement in this systematic review. The present case study was not specifically funded. J.C., P.G., and C.D.M. were involved in the development of systematic review automation tools used in the systematic review.

: Conflict of interest: None.

View full text

Original ArticleA full systematic review was completed in 2 weeks using automation tools: a case study

Abstract

Background and Objectives

Study Design and Setting

Results

Conclusion

Section snippets

Background

The systematic review

Time to complete the full systematic review

Discussion

Conclusion

Acknowledgments

J Clin Epidemiol

Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry

BMJ Open

Making progress with the automation of systematic reviews: principles of the international collaboration for the automation of systematic reviews (ICASR)

Syst Rev

Moving toward the automation of the systematic review process: a summary of discussions at the second meeting of International Collaboration for the Automation of Systematic Reviews (ICASR)

Syst Rev

Usage of automation tools in systematic reviews

Res Synth Methods

Increased fluid intake to prevent urinary tract infections: systematic review and meta-analysis

Br J Gen Pract

Original Article
A full systematic review was completed in 2 weeks using automation tools: a case study