*Important notice: medRxiv publishes preliminary scientific reports that have not been peer-reviewed and therefore should not be considered conclusive, guiding clinical practice/health-related behavior, or treated as established information.
In a recent study published on the preprint server medRxiv*, researchers in Australia, Finland and New Zealand reviewed the translational success of artificial intelligence (AI) models in healthcare, specifically those used in studies on coronavirus disease 2019 (COVID-19) were used.
Study: Applying a Comprehensive Assessment Framework to Coronavirus Disease Studies 19: A Systematic Review of the Translational Aspects of Artificial Intelligence in Healthcare. Photo credit: sdecoret / Shutterstock.com
background
Although some AI applications are in clinical trials to determine their potential integration into medical information systems, studies demonstrating their ability to improve clinical outcomes are still lacking. However, studies have shown the superiority of AI in experimental or pilot environments. Due to the reduced performance of these AI applications with external validation and low acceptance by clinicians, existing clinical workflows have yet to initiate their integration.
About the study
In the present study, researchers evaluate COVID-19 AI models developed between December 2019 and 2020 using the Translational Assessment of AI in Healthcare (TEHAI), a comprehensive assessment framework for translational value assessments of AI models.
TEHAI evaluates scientific studies for acceptability, inherent capabilities and usefulness to focus on the statistics of its 15 sub-components. This expert-led formalized framework softens the subjectivity of an individual and replaces it with the consensus power of multiple reviewers. Each criterion of this framework resulted in a score between zero and three points, depending on study quality.
This systematic review used the Covidence software platform. While nine independent reviewers rated the scholarly literature for translation value, the other two collected descriptive data from each study. A third reviewer compared the assessment results and extracted data from all studies to resolve any discrepancies.
Fisher’s exact test was used to assess associations between groupings of academic papers and the distributions of subcomponent scores. Finally, Kendall’s formula was used to calculate associations between all 15 subcomponents.
Results
Screening over a one-year period revealed over 3,000 eligible studies, indicating high activity in this area. However, only 102 studies produced the expected results.
Most studies achieved remarkable performance in the capability component, but did not score highly in the utility and service acceptance components of the TEHAI framework.
Most studies scored high for technical performance but low for clinical applicability. However, most studies also failed due to AI-related parameters such as ethics, safety, validation of external models and quality of integration with medical systems.
69 of the 102 studies related to medical image analysis, with a convolutional neural network being the most popular machine learning mode. This result was expected since imaging techniques are now well known and readily applied in real-world clinical settings. However, non-imaging studies performed better on the acceptability and usefulness subcomponents.
Surprisingly, studies with large datasets have yielded no gains in terms of utility or acceptability. This was also expected as the number of studies to be analyzed would increase and the differences between small and large datasets would also become significant.
Few independent studies have tested the claim that AI models provide more accurate and specific real-time results than human experts. Therefore, despite their potential, AI models are generally not suitable for clinical translation and could lead to undesirable results if deployed prematurely. Some side effects could be an increased burden on the healthcare system and patients with redundant invasive procedures that could lead to deaths due to misdiagnosis.
Most of the studies required more appropriate considerations for the service acceptance area of the TEHAI framework, which was related to the real-world applications of AI-based models in the medical industry. Therefore, more pilot data from real-world testing with AI-based new tools is needed to adjust costs through misclassification and deployment from a patient safety perspective. A provisional settlement of the labor requirement is also urgently required.
Conclusions
The present review evaluated 102 COVID-19 AI studies to reveal a notable gap in most studies that could adversely affect their clinical translation. These results underscore the importance of addressing the challenge of AI translatability in the field of medical information systems.
Researchers should also introduce appropriate interventions early in the AI development cycle to improve translatability. In this respect, the TEHAI assessment framework could be helpful. In addition, the results from its application could inform all stakeholders including developers, researchers and clinicians to deploy more translatable AI models in healthcare.
*Important notice: medRxiv publishes preliminary scientific reports that have not been peer-reviewed and therefore should not be considered conclusive, guiding clinical practice/health-related behavior, or treated as established information.
Written by
Neha Mathur
Neha is a digital marketing expert based in Gurugram, India. She holds a Masters degree from University of Rajasthan with specialization in Biotechnology in 2008. She has experience in pre-clinical research through her research project in the Department of Toxicology at the renowned Central Drug Research Institute (CDRI), Lucknow, India. She also has a certification in C++ programming.
quotes
Please use one of the following formats to cite this article in your essay, paper, or report:
APA
Mathur, Neha. (2023, March 01). A lack of studies limit the integration of artificial intelligence into the clinic. News Medical. Retrieved March 01, 2023 from https://www.news-medical.net/news/20230301/Lack-of-studies-limit-the-integration-of-artificial-intelligence-into-the-clinic.aspx.
MLA
Mathur, Neha. “Lack of studies limit the integration of artificial intelligence into the clinic”. News Medical. March 1, 2023.
Chicago
Mathur, Neha. “Lack of studies limit the integration of artificial intelligence into the clinic”. News Medical. https://www.news-medical.net/news/20230301/Lack-of-studies-limit-the-integration-of-artificial-intelligence-into-the-clinic.aspx. (accessed on 03/01/2023).
Harvard
Mathur, Neha. 2023. A lack of studies limits the integration of artificial intelligence into the clinic. News-Medical, accessed on March 1, 2023, https://www.news-medical.net/news/20230301/Lack-of-studies-limit-the-integration-of-artificial-intelligence-into-the-clinic. aspx .
