The number arrives looking like a fact. A brain MRI goes through a volumetric software package, and a few minutes later the report carries a hippocampal volume with a percentile attached: this person's memory structures, ranked against a reference population. In a memory clinic that percentile can tip a conversation one way or the other, toward reassurance or toward further testing. The quiet question underneath it is simple and, until recently, largely unasked in public: ranked against which population? Twenty of the 24 artificial intelligence devices the Food and Drug Administration has authorized for Alzheimer's disease and related dementias do exactly this kind of measurement, and a team of researchers has now gone looking for the people behind the math.
The audit, a cohort study published as a research letter in JAMA on July 30, 2025 by Krista Chen and colleagues at Johns Hopkins, Yale and the University of Pennsylvania, took the FDA's own records as its raw material: the agency's AI and machine learning enabled device list, the 510(k) database, the De Novo database, plus company websites and PubMed for any peer-reviewed paper describing the algorithms. They found 24 devices authorized between January 2015 and December 2024, with 10 of them clearing in the final three years. Twenty-two went through the 510(k) pathway, which turns on showing substantial equivalence to a device already on the market rather than fresh clinical proof; two came through De Novo. Twenty-one were reviewed by radiology panels, three by neurology. Median time to authorization was 228 days. Beyond the volumetric tools, three were functional tests of memory, learning or visuospatial awareness, and two quantified amyloid deposition on PET. Every one of them is labeled for clinician use only.
Then the researchers asked what the public record says about the data that taught these systems. FDA summaries described training data for 10 devices (41.7 percent) and validation data for just 2 (8.3 percent). Where datasets were described at all, they were mostly retrospective and multicenter, with mean sample sizes running between 261 and 845 patients, according to the companion abstract the same team presented at the Alzheimer's Association International Conference and published in Alzheimer's & Dementia on December 23, 2025. Disease status, meaning which dementia the patients actually had and in what proportions, along with age and sex, was reported for fewer than half the devices. Race and ethnicity were, in the authors' phrasing, rarely disclosed: the conference abstract puts it at a maximum of one device from each reporting source. AuntMinnie, reporting from the conference session, noted that for 12 of the 24 devices the team found no information about training or validation datasets in either the FDA summaries or the published literature.
It is worth being precise about what this study is and is not. It is an audit of documents, not a test of machines. Nobody ran these algorithms on a diverse patient sample and measured where they slipped. No device in this analysis was shown to misread a scan from a Black patient, or a woman, or someone of 88. What the study establishes is narrower and, in its way, more unsettling: that a clinician holding the output cannot look up the answer. The reference population behind the percentile is, for most of these products, simply not in the public file. Dementia is not an equal-opportunity illness in the United States, and brain structure varies with age, sex, vascular health and much else, which is why the authors frame the gap as a question of generalizability rather than a finding of harm.
For a family sitting in the clinic, this changes less about the appointment than it might seem and more about how to read the room. These are decision-support tools, not deciders. A volumetric report does not diagnose anyone; it sits alongside the history, the examination, the cognitive testing and the clinician's judgment, which is precisely why the FDA restricted all 24 to professional use. What the audit gives families is a legitimate question to put to the person reading the scan: how much weight is this software carrying in your thinking, and what is known about the patients it learned from. That is a conversation for the consulting room, with the clinician who ordered the test.
The regulatory backdrop is moving, slowly. The FDA, with international counterparts, issued guiding principles in 2021 encouraging demographic representativeness in machine learning devices. On January 7, 2025 the agency published draft guidance on AI-enabled device software functions that proposes recommendations for demographic data reporting, or a justification when such data are unavailable. Draft is the operative word: the Federal Register notice states plainly that the document is not final and not for implementation. The authors list their own constraints too, including inconsistent reporting in approval documents and the possibility that devices missing from the databases, or validation studies missing from the literature, escaped the search entirely.