The U.S. Food and Drug Administration has established guiding principles for transparency in artificial intelligence and machine learning (AI/ML) medical devices, aiming to ensure their safety and promote health equity. However, independent analyses of publicly available data reveal a significant gap between these regulatory ideals and the information manufacturers provide. Research shows that crucial data on patient demographics and device safety is consistently underreported for hundreds of FDA-approved AI products, creating a substantial risk of algorithmic bias that could amplify health disparities and compromise patient care.

The FDA's Stated Principles for AI/ML Device Transparency

The FDA’s Center for Devices and Radiological Health (CDRH) regulates AI-enabled medical devices under the Federal Food, Drug, and Cosmetic Act, employing a risk-based approach that considers a device's intended use and technological characteristics. A central pillar of this oversight is transparency, which the agency defines as the degree to which appropriate information about a device—including its intended use, development, performance, and logic—is clearly communicated to relevant audiences. According to the FDA, this clarity is essential for safe and effective use, as it supports informed decision-making, helps identify bias, and ultimately promotes health equity.

The agency’s principles call for clear documentation of an algorithm's design, the data used to train it, and its decision-making logic, in line with what it calls Good Machine Learning Practices (GMLP). During a public workshop on the topic, stakeholders including patients, providers, and software developers recommended a focus on the data sources used to train and validate AI software. They specifically urged the FDA and manufacturers to provide demographic information about the populations involved in testing. This aligns with the FDA's own stated concerns that algorithms trained on data with limited patient and site diversity may not generalize well to clinical practice across the United States.

The FDA's expectations extend across the "total product life cycle," not just the initial approval. This means developers are expected to monitor real-world performance and have a predetermined change control plan to manage updates without needing a full new submission for every modification. This regulatory flexibility is intended to support innovation while maintaining safety. A key part of this ongoing oversight is the principle that users, such as clinicians, should receive clear and essential information. As the FDA states, transparency supports "health equity through identification of bias" and increases "confidence in MLMD use."

FDA AI/ML Device Transparency: Principles vs. Reality

Despite the FDA's clear guidance, a review of publicly available data for approved devices reveals a consistent failure by many manufacturers to provide the information necessary to assess fairness and safety. The following table contrasts the agency's principles with the documented reality found in recent independent research.

A comparison of FDA principles for AI/ML device transparency against the documented reality of manufacturer reporting and the resulting risks.
FDA Principle for Transparency Documented Reality (from Research) Consequence/Risk
Training Data Characterization The FDA emphasizes transparency regarding data sources, bias in training data, and the inclusion of diverse populations to ensure algorithms are robust and generalizable. An analysis as of December 2024 found "significant gaps" between FDA guidance and publicly available data. A separate scoping review of 950 approved devices found that demographic and socioeconomic characteristics are consistently underreported in approval documents. Underrepresentation in training data can amplify existing racial and gender disparities. This can lead to flawed models that produce incorrect scoring or diagnoses, thereby exacerbating health inequities.
Post-Market Performance & Safety The FDA expects manufacturers to have change-management plans and to monitor real-world performance to ensure continued safety and effectiveness, reporting on "device modifications and updates." Researchers found that FDA reporting data remains inconsistent, with irregular reporting of safety considerations and potential adverse events in device approval documents. Without consistent safety and performance data, it is difficult for regulators and healthcare providers to monitor for "model drift" or performance degradation, increasing risks to patient safety.

The Concrete Risks of Opaque AI in Medicine

The gaps in data transparency for AI-enabled medical devices are not merely administrative shortcomings; they create tangible risks for patients and health systems. AI models can be flawed due to problems with the data used to develop them. When training datasets are not representative of the diverse populations on which the devices will be used, these flaws can perpetuate and even worsen existing health inequities.

Research shows that the underreporting of demographic characteristics in FDA submissions directly exacerbates the risk of algorithmic bias. An AI tool trained predominantly on data from one demographic group may underperform when used on patients from other groups, leading to inaccurate diagnoses or risk scores. This can result in what one device's safety report described as the "misuse of the device to analyze images from an unintended patient population." That device, the Brainomix 360 e-ASPECTS, also highlighted potential risks of incorrect scoring by the algorithm and outright device failure. Without consistent reporting, it is difficult to know how many other devices carry similar unstated risks.

This lack of transparency also places a significant burden on healthcare providers. The FDA advises that AI performance should be monitored in the specific environment where it is being used. However, as a study in npj Digital Medicine notes, this is "a difficult aim to implement without transparency on how the model was tested and evaluated." Without access to the underlying data on a device's validation, clinicians cannot confidently assess whether a tool is appropriate for their specific patient population, undermining their ability to ensure safe and equitable care. They are left to trust a "black box" tool without the information needed to verify its local accuracy.

Moving Towards Equitable AI in Medical Devices

Healthcare providers, policymakers, and device developers should advocate for and implement mandatory, standardized reporting of demographic and safety data for AI/ML medical devices. Researchers analyzing the gaps in current reporting argue that regulatory bodies like the FDA should push for a mandatory release of safety data and detailed information on potential adverse events. This would shift the burden of proof to manufacturers to demonstrate their products are safe and effective across diverse populations.

Furthermore, to safeguard against overly optimistic performance claims based on narrow datasets, experts recommend that developers conduct "prospective designs and preregistered external, multicenter validation" before a device is approved. Such robust validation across diverse settings and populations is a critical step toward building trust and ensuring that AI tools benefit all patients, not just a select few. Early and frequent engagement between developers and the FDA throughout the design process could also help identify and mitigate potential biases before a product reaches the market.

Ultimately, the success of these measures will be determined by a clear, observable outcome: an increased availability of detailed demographic data in public FDA approval documents and post-market surveillance reports for AI/ML medical devices. Only with this level of transparency can the full potential of AI in medicine be realized safely and equitably for everyone.

Sources