m
Recent Posts
Connect with:
Thursday / November 19.
HomeminewsStudy Finds Gaps in Human Oversight of Healthcare AI

Study Finds Gaps in Human Oversight of Healthcare AI

While health professionals often help develop artificial intelligence (AI) tools for clinical care, less is known about how they remain involved in monitoring these systems once in use, Australian research has found. 

New research led by Flinders University and other experts contends that keeping healthcare AI aligned with clinical needs requires ongoing human input. 

The approach – often called “human in the loop” – refers to how people contribute to the development, checking, overseeing, and updating of an AI system. 

However, the study found that this term was used inconsistently, making it difficult to understand what people actually did, and how their involvement affected the AI system. 

“We found much more detail about how clinicians helped build and check AI tools than about how they oversee them during everyday care.”

Published in the International Journal of Medical Informatics,1 the study examines how clinicians contribute to AI systems that support clinical decisions, and proposes a checklist to help describe this involvement more clearly. 

The review of 12 studies variously described clinician involvement during AI tool development, such as labelling clinical information or helping design rules. Eleven of the studies described involvement in reviewing the AI system, two described maintenance, and none reported formal processes for ongoing human involvement. 

Lead author of the review article, Michael Bakker, a Flinders University PhD candidate, said the findings highlight a need to explain the importance of continuing clinical expertise throughout an AI tool’s use. 

“We found much more detail about how clinicians helped build and check AI tools than about how they oversee them during everyday care. 

“We need to understand how clinicians can flag problems and how that feedback leads to changes in the system.” 

Co-author Professor Niranjan Bidargaddi, who leads the Digital Health Research Lab at Flinders University, said the findings point to a major research gap in “how we keep clinical AI safe and useful once it enters everyday care”. 

“The TGA (Therapeutic Goods Administration) and international regulators are increasingly focused on monitoring AI medical devices throughout their use. The challenge now is to establish how this works in real clinical settings, including who checks performance, who identifies problems, and who is responsible for acting on them. 

“Clinical expertise needs to remain part of the process throughout an AI tool’s use, with oversight built into routine clinical work,” Prof Bidargaddi said. 

The team developed a reporting checklist, asking researchers to explain who is involved in AI tools, how they contribute, why their input is needed, and what difference it makes. It also asks them to describe the limits of their involvement and provide supporting documentation. 

The checklist is intended to make human involvement easier to understand and assess across an AI tool’s development and use. It provides a starting point for clearer reporting and further research into ongoing safety of health care AI.

Reference 

  1. Bakker M, Van Garderen A, Bidargarddi N, et al Human in the loop in AI-enabled clinical decision support: a systematic scoping review and reporting checklist for lifecycle governance. Int J Med Inform. 2026 Dec;222:106701. doi: 10.1016/j.ijmedinf.2026.106701.  

DECLARATION

DISCLAIMER : THIS WEBSITE IS INTENDED FOR USE BY HEALTHCARE PROFESSIONALS ONLY.
By agreeing & continuing, you are declaring that you are a registered Healthcare professional with an appropriate registration. In order to view some areas of this website you will need to register and login.
If you are not a Healthcare professional do not continue.