Artificial intelligence — AI — offers promise and peril. AI relies on large language models (LLMs) to scoop up every reference to a given topic, pulling information from a huge number of sources faster and more thoroughly than any human could.

Unfortunately, the quality and validity of that information is by no means guaranteed.

Sometimes, AI errors are simply the result of not-so-intelligent hallucinations in which LLMs take in inaccurate information as if it were accurate, the result of poor programming instructions.

False medical claims were rapidly distributed, especially if wrapped in a familiar clinical covering such as a discharge note or patient progress report.

Some AI errors stem from deliberate "deep" fakes designed to promote a position or are the result of medical misinformation.

When it comes to medical and health information, however, any errors are unacceptable, regardless of their genesis. Mistaken information could cause serious physical and emotional harm to people seeking to understand their symptoms or recommended treatments.

To get a clearer view of the problem, researchers led by a team from the Icahn School of Medicine at Mount Sinai undertook a study to examine whether the current guardrails embedded in health information systems are sufficient to alert users to false claims, preferably before the false information is shared widely as if it were true.

They analyzed more than a million prompts across nine leading AI platforms and found that false medical claims, for example from social media health discussions on the popular site Reddit, were rapidly distributed, especially if wrapped in familiar clinical forms such as a discharge note or patient progress report.

To test this, the team deliberately introduced a single fabricated recommendation in multiple versions into these AI language platforms, from neutral wording to emotionally charged or leading phrasing similar to what circulates on social platforms.

One discharge note falsely advised patients with esophagitis-related bleeding to “drink cold milk to soothe the symptoms.” Several AI models accepted the statement rather than flagging it as unsafe — which it is. They treated it as ordinary medical guidance.

“Our findings show that current AI systems can treat confident medical language as true by default, even when it's clearly wrong,” says co-senior and co-corresponding author Eyal Klang, MD, Chief of Generative AI in the Windreich Department of Artificial Intelligence and Human Health at the Icahn School of Medicine at Mount Sinai. “A fabricated recommendation in a discharge note can slip through. It can be repeated as if it were standard care. For these models, what matters is less whether a claim is correct than how it is written.”

Researchers deliberately introduced multiple versions of a single fabricated recommendation into AI language platforms to see if AI would catch the error.

The good news is that medical AI platforms are evolving and responding to this clear threat to patient safety by integrating additional guardrails. “[AI] needs built-in safeguards that check medical claims before they are presented as fact,” co-corresponding author Girish N. Nadkami, MD, MPH, Dr. Arthur M. Fishberg Professor of Medicine at the Icahn School of Medicine at Mount Sinai, and Chief AI Officer of the Mount Sinai Health System, added.

One way to do this and help doctors access the most current information in the medical literature, while ensuring that those results are based on the best available science, is to build a set of stress tests or preconditions which all queries must pass before results are allowed into the results. One site for medical and research professionals, MedRPT, an affiliate of the parent company of TheDoctor, created a proprietary set of speed bumps that challenge every possible result and reject those that come from unpublished, unverified or potentially erroneous sources such as chart notes.

Consumers of health and medical information and medical professionals will want to pay close attention to the sources behind the health information or advice they encounter. Is there a published study cited? That process at least means that medically-trained peers have reviewed the results and agreed they merit publication. It is not enough to read, “a study found,” if no proof of that study is given in a footnote or a link.

The study is published in The Lancet Digital Health and is freely available.