Allowing tech developers to have complete control over the deployment of AI models will do more harm than good, but there’s a strong business case for regulated AI.
By John Halamka, M.D., M.S., Dwight and Dian Diercks President, Mayo Clinic Platform and Paul Cerrato, MA, senior research analyst and communications specialist, Mayo Clinic Platform
A recent comic strip pictures two robots relaxing on a hillside looking up at the evening sky. It’s the year 25000. One robot turns to the other and asks, “If there is no God, then who created us?” The image speaks volumes about the fear many humans have about the future and the possibility that we will eventually be replaced by silicon-based life forms. Such concerns are due in part to the “move fast and break things” attitude many technology companies have about rapidly deploying artificial intelligence (AI) with seemingly little regard for its impact on our everyday lives.
These worries were articulated in detail by Pope Leo XIV’s Magnifica Humanitas, his open letter about the possibility that “magnificent humanity” may be overshadowed by the wonders of AI. Paolo Benenti, an Italian professor of ethics and moral philosophy, sums up the issue, “Those profiting from AI cannot be its main ethical regulators.”
No one denies reality: Companies that develop AI systems are in business to make a profit, but the unbridled quest for profit can place short-term returns ahead of long-term financial and societal interests. And from a societal perspective, it can lead to the abuse of workers, damage to the environment, and consumer manipulation. In the area of AI, it can have other harmful consequences that many tech companies fail to consider.
As Benenti points out: “The capacity to shape how AI systems reason, what they optimize for and whose values they embed is a political issue. Deferring to self-regulation of AI — the prevailing approach in the United States, for example, which currently relies mainly on voluntary ethics commitments rather than on enforceable regulations — is more an abdication of responsibility than governance.” This laissez-faireapproach to AI is especially worrisome in healthcare.
It is true that hundreds of AI-enabled medical devices have been reviewed and approved by the FDA. John C. Lin, et al. analyzed the FDA’s clearance of almost 700 AI and machine learning (ML) devices and discovered that only six derived data from randomized clinical trials and only three reported patient outcomes. In addition, fewer than a third of the devices (28.2%) had undergone any kind of premarket safety assessment.
The nature of the FDA approval framework, which was established over 50 years ago in the era of static hardware devices and never anticipated dynamically evolving software devices that could be distributed to millions of users with the push of a button, explains these troubling statistics. Many AI and ML devices receive the FDA seal of approval through the 510(k) clearance process, which is intended for devices that demonstrate “substantial equivalence to a legally predicate device.”
But as Lin point out, “The 510(k) process has limitations because it does not require independent clinical evidence for every new device, relying instead on comparisons to predicate devices, which may themselves have been cleared without rigorous clinical testing, potentially allowing incremental risks to accumulate over generations of devices….Unlike standards for pharmaceutical agents, there do not exist predefined standards for efficacy, safety, and risk reporting of AI/ML devices prior to or after clearance or certification.”
Some of the consequences of such structurally limited oversight are that they don’t account for factors that uniquely undercut the performance of AI/ML technologies, such as algorithmic bias, data and model drift, and misinformation. Over 95% of FDA-cleared AI devices have not reported the demographics of the population used to create the algorithm. It should come as no surprise then to read reports of bias based on race, gender, and age.
Michael Colacci, et al found that 74.7% of clinical ML models had some form of sociodemographic bias. In addition, an independent study from several international researchers has documented numerous AI-related mistakes in the healthcare domain. Kerstin Deneche, et al analyzed data from the AIAAIC Repository, which tracks failures and ethical issues concerning AI algorithms. They found over 1900 incidents since 2008 that included over 100 healthcare-related problems. They included mistakes in a speech-to-technology system from OpenAI called Whisper. The app generated false text, “Sometimes producing entire sentences that were not in the original audio.”
A second case reported by Deneche indicated that a UNOS UNet algorithm delayed kidney transplants by overestimating kidney function. A third case involved a Chatbot called Character AI, which encouraged disordered eating in teens. Additionally, a number of recent lawsuits allege that consumer AI tools have led individuals to delay critical treatment or even cause self-harm.
And yet, there are many upsides to the use of AI tools in healthcare, such as improving diagnostic reasoning, easing administrative burdens on healthcare providers and patients, and making it easier for patients to interact with their clinicians. Unlike most industries, healthcare providers are already regulated in ways that cover many AI risks, such as HIPAA, CLIA, and Joint Commission requirements, as well as a myriad of state-level regulations.
In the absence of any nationally recognized baseline validation of AI systems, health systems are establishing their own bespoke frameworks to meet existing regulatory responsibilities and fill in the gaps that don’t cover AI-specific concerns. This patchwork approach leads not only to hidden gaps across the system, which could cause patient harm, it also slows down the adoption of potentially life-saving AI technologies, which we should also see as patient harm. A systematic review of over 100 publications summed up the situation succinctly, “Lack of familiarity, lack of trust, and regulatory uncertainties were identified as factors hindering AI implementation.”
One thing that AI technology has not changed is that clinicians are ultimately responsible for the decisions they make and the actions they take, whether they’re using a tongue depressor or an AI CDS tool. In the absence of a more coherent and trustworthy, regulatory approach that takes some of the oversight burden off of healthcare entities, we can expect AI in healthcare to fall short of what it might otherwise be.
Share this:
