A new study by Nanyang Technological University, Singapore (NTU Singapore) has found that many developers building medical AI tools lack familiarity with regulatory frameworks.
The study, one of the first multi-regional surveys of medical AI developers focused specifically on regulatory awareness, also revealed that most developers believe they are responsible for the AI they build.
Reported in npj Digital Medicine, the findings indicate a stronger need to engage AI developers to boost the effectiveness and safety of implementing AI in medical settings, especially in countries like Singapore that invest heavily in digital innovation.
Limited familiarity of medical AI regulations a stumbling block
As AI continues to be increasingly used in healthcare, such as to analyse medical images and predict disease risks, limited awareness of regulatory safeguards such as the EU AI Act or Singapore’s Artificial Intelligence in Healthcare Guidelines could affect patient outcomes and public trust when AI tools are deployed in the clinic.
The researchers surveyed 122 medical AI developers from Singapore and other regions such as China, Hong Kong and the United Kingdom to find out their awareness, familiarity and perceptions of implementing these regulations.
They found that most developers felt they should be responsible for the AI they create, with guidance from a government or institutional ethics body, and in partnership with users and the organisations that provided the data for the development of the AI.
More than half of the surveyed developers (57 per cent) were aware of at least one regulatory framework, with senior developers and developers from non-academic organisations more likely to be more aware of more frameworks than junior developers and developers from academic organisations. Furthermore, senior developers were also more familiar with these frameworks.
However, more than half (67 per cent) of the organisations that the developers worked for have not adopted any regulatory frameworks. Developers from organisations that adopted these frameworks were more familiar with the frameworks compared to those from non-adopting organisations.
The researchers say addressing these gaps will be key to ensuring AI can be effectively and safely integrated into healthcare systems.
“Developers are uniquely positioned to assess data quality, model limitations and risks such as bias and hallucinations. While it is reassuring that there is a sense of responsibility among developers for the AI tools they create, the gaps in regulatory awareness identified in our study are concerning,” said Asst Prof Wilson Goh of NTU’s Lee Kong Chian School of Medicine (LKCMedicine) and Chief Data Scientist at the Centre of AI in Medicine (C-AIM), who co-led the study.
“A more proactive stance is needed to empower developers to build high-quality AI models that do not pose a risk to patient outcomes and public trust.”
Implementing trustworthy AI in the clinic
To increase awareness of AI regulations, the researchers propose that educational institutions include regulatory frameworks in developer training. Within companies and development teams, well-structured mentorship processes between senior and junior developers can also reduce the difficulties of navigating the complex regulatory landscape.
Beyond improving organisational culture and leadership to increase awareness and familiarity of frameworks, the researchers recommend that national regulatory bodies adopt an active role in ensuring compliance as well as implement longer-term harmonisation activities across jurisdictions, incorporating both positive and negative experiences from developers.
These combined efforts can potentially ease adoption barriers. The findings from this study complement an earlier study on medical AI perceptions by the researchers that shows gastroenterologists generally trust and accept the use of AI medical tools in clinics and hospitals.
“Our research shows that collaborative efforts from multiple stakeholders are required to resolve the challenges of adopting AI in healthcare. Only by taking collective ownership of the broader implications of AI deployment will we be able to address the challenges of ensuring patient safety, data privacy and the long-term reliability of AI in medicine,” says NTU Senior Vice President (Health & Life Sciences) and Director of C-AIM Prof Joseph Sung, who led the study. Prof Sung is also Dean of LKCMedicine.
