Randomized clinical trials of machine learning interventions in health care: a systematic review

Importance: Despite the potential of machine learning to improve multiple aspects of patient care, barriers to clinical adoption remain. Randomized clinical trials (RCTs) are often a prerequisite to large-scale clinical adoption of an intervention, and important questions remain regarding how machin...

Full description

Saved in:
Bibliographic Details
Main Authors: Plana, Deborah, Shung, Dennis L., Grimshaw, Alyssa A., Saraf, Anurag, Sung, Joseph Jao Yiu, Kann, Benjamin H.
Other Authors: Lee Kong Chian School of Medicine (LKCMedicine)
Format: Article
Language:English
Published: 2023
Subjects:
Online Access:https://hdl.handle.net/10356/164627
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Nanyang Technological University
Language: English
id sg-ntu-dr.10356-164627
record_format dspace
spelling sg-ntu-dr.10356-1646272023-03-05T16:54:44Z Randomized clinical trials of machine learning interventions in health care: a systematic review Plana, Deborah Shung, Dennis L. Grimshaw, Alyssa A. Saraf, Anurag Sung, Joseph Jao Yiu Kann, Benjamin H. Lee Kong Chian School of Medicine (LKCMedicine) Science::Medicine Clinical Decision Making Machine Learning Importance: Despite the potential of machine learning to improve multiple aspects of patient care, barriers to clinical adoption remain. Randomized clinical trials (RCTs) are often a prerequisite to large-scale clinical adoption of an intervention, and important questions remain regarding how machine learning interventions are being incorporated into clinical trials in health care. Objective: To systematically examine the design, reporting standards, risk of bias, and inclusivity of RCTs for medical machine learning interventions. Evidence Review: In this systematic review, the Cochrane Library, Google Scholar, Ovid Embase, Ovid MEDLINE, PubMed, Scopus, and Web of Science Core Collection online databases were searched and citation chasing was done to find relevant articles published from the inception of each database to October 15, 2021. Search terms for machine learning, clinical decision-making, and RCTs were used. Exclusion criteria included implementation of a non-RCT design, absence of original data, and evaluation of nonclinical interventions. Data were extracted from published articles. Trial characteristics, including primary intervention, demographics, adherence to the CONSORT-AI reporting guideline, and Cochrane risk of bias were analyzed. Findings: Literature search yielded 19737 articles, of which 41 RCTs involved a median of 294 participants (range, 17-2488 participants). A total of 16 RCTS (39%) were published in 2021, 21 (51%) were conducted at single sites, and 15 (37%) involved endoscopy. No trials adhered to all CONSORT-AI standards. Common reasons for nonadherence were not assessing poor-quality or unavailable input data (38 trials [93%]), not analyzing performance errors (38 [93%]), and not including a statement regarding code or algorithm availability (37 [90%]). Overall risk of bias was high in 7 trials (17%). Of 11 trials (27%) that reported race and ethnicity data, the median proportion of participants from underrepresented minority groups was 21% (range, 0%-51%). Conclusions and Relevance: This systematic review found that despite the large number of medical machine learning-based algorithms in development, few RCTs for these technologies have been conducted. Among published RCTs, there was high variability in adherence to reporting standards and risk of bias and a lack of participants from underrepresented minority groups. These findings merit attention and should be considered in future RCT design and reporting. Published version This study was supported by grants K23-DK125718 (Dr Shung) and K08-DE030216 (Dr Kann) from the National Institutes of Health, grant T32GM007753 from the National Institute of General Medical Sciences (Ms Plana), and grant F30-CA260780 from the National Cancer Institute (Ms Plana). 2023-02-07T00:59:17Z 2023-02-07T00:59:17Z 2022 Journal Article Plana, D., Shung, D. L., Grimshaw, A. A., Saraf, A., Sung, J. J. Y. & Kann, B. H. (2022). Randomized clinical trials of machine learning interventions in health care: a systematic review. JAMA Network Open, 5(9), e2233946-. https://dx.doi.org/10.1001/jamanetworkopen.2022.33946 2574-3805 https://hdl.handle.net/10356/164627 10.1001/jamanetworkopen.2022.33946 36173632 2-s2.0-85139162395 9 5 e2233946 en JAMA Network Open © 2022 Plana D et al. This is an open access article distributed under the terms of the CC-BY License. application/pdf
institution Nanyang Technological University
building NTU Library
continent Asia
country Singapore
Singapore
content_provider NTU Library
collection DR-NTU
language English
topic Science::Medicine
Clinical Decision Making
Machine Learning
spellingShingle Science::Medicine
Clinical Decision Making
Machine Learning
Plana, Deborah
Shung, Dennis L.
Grimshaw, Alyssa A.
Saraf, Anurag
Sung, Joseph Jao Yiu
Kann, Benjamin H.
Randomized clinical trials of machine learning interventions in health care: a systematic review
description Importance: Despite the potential of machine learning to improve multiple aspects of patient care, barriers to clinical adoption remain. Randomized clinical trials (RCTs) are often a prerequisite to large-scale clinical adoption of an intervention, and important questions remain regarding how machine learning interventions are being incorporated into clinical trials in health care. Objective: To systematically examine the design, reporting standards, risk of bias, and inclusivity of RCTs for medical machine learning interventions. Evidence Review: In this systematic review, the Cochrane Library, Google Scholar, Ovid Embase, Ovid MEDLINE, PubMed, Scopus, and Web of Science Core Collection online databases were searched and citation chasing was done to find relevant articles published from the inception of each database to October 15, 2021. Search terms for machine learning, clinical decision-making, and RCTs were used. Exclusion criteria included implementation of a non-RCT design, absence of original data, and evaluation of nonclinical interventions. Data were extracted from published articles. Trial characteristics, including primary intervention, demographics, adherence to the CONSORT-AI reporting guideline, and Cochrane risk of bias were analyzed. Findings: Literature search yielded 19737 articles, of which 41 RCTs involved a median of 294 participants (range, 17-2488 participants). A total of 16 RCTS (39%) were published in 2021, 21 (51%) were conducted at single sites, and 15 (37%) involved endoscopy. No trials adhered to all CONSORT-AI standards. Common reasons for nonadherence were not assessing poor-quality or unavailable input data (38 trials [93%]), not analyzing performance errors (38 [93%]), and not including a statement regarding code or algorithm availability (37 [90%]). Overall risk of bias was high in 7 trials (17%). Of 11 trials (27%) that reported race and ethnicity data, the median proportion of participants from underrepresented minority groups was 21% (range, 0%-51%). Conclusions and Relevance: This systematic review found that despite the large number of medical machine learning-based algorithms in development, few RCTs for these technologies have been conducted. Among published RCTs, there was high variability in adherence to reporting standards and risk of bias and a lack of participants from underrepresented minority groups. These findings merit attention and should be considered in future RCT design and reporting.
author2 Lee Kong Chian School of Medicine (LKCMedicine)
author_facet Lee Kong Chian School of Medicine (LKCMedicine)
Plana, Deborah
Shung, Dennis L.
Grimshaw, Alyssa A.
Saraf, Anurag
Sung, Joseph Jao Yiu
Kann, Benjamin H.
format Article
author Plana, Deborah
Shung, Dennis L.
Grimshaw, Alyssa A.
Saraf, Anurag
Sung, Joseph Jao Yiu
Kann, Benjamin H.
author_sort Plana, Deborah
title Randomized clinical trials of machine learning interventions in health care: a systematic review
title_short Randomized clinical trials of machine learning interventions in health care: a systematic review
title_full Randomized clinical trials of machine learning interventions in health care: a systematic review
title_fullStr Randomized clinical trials of machine learning interventions in health care: a systematic review
title_full_unstemmed Randomized clinical trials of machine learning interventions in health care: a systematic review
title_sort randomized clinical trials of machine learning interventions in health care: a systematic review
publishDate 2023
url https://hdl.handle.net/10356/164627
_version_ 1759856527390277632