AI tools used in Ontario doctor’s offices are hallucinating inaccurate patient details
The finding is part of a new report on AI from the province’s auditor general
A new report from Ontario’s auditor general indicates that artificial intelligence note-taking tools used by some doctors in the province have recorded patient information incorrectly, with some even having AI hallucinations, a term used to describe fabrications provided by large-language models.
The findings were published as part of a broader look at the provincial public service’s use of AI.
“Inaccuracies in medical notes generated by AI scribe systems could potentially result in inadequate or harmful treatment plans that may potentially impact patient health outcomes,” the report said.
The results of the report have frightening implications, with 12 of 20 programs that were analyzed containing false information, such as incorrectly named prescribed medications. Seventeen of the 20 missed important information regarding patient mental health.
Minister of Public and Business Service Delivery and Procurement, Stephen Crawford, said the programs are used to save physicians time during appointments. “That gives them more time to spend with their patients and less time in record keeping,” he said.
While there’s no doubt that doctors and nurses in Ontario are severely under-resourced, the troubling report also notes security risks related to AI use and personal data.
“The Ministry had not implemented security controls to prevent Ontario Public Service (OPS) staff from inadvertently uploading Ontarians’ personal information, such as their health cards, driver’s licences and credit card information, or sensitive corporate data, such as vendor contracts and invoices, onto these Al websites,” it said. “When OPS staff use publicly available GenAl websites, there is a risk that these websites can retain and use the data or any personal or sensitive information entered by staff to train the sites large-language model software.”
Related: I spilled my secrets to a ChatGPT therapist. It didn’t go well
Carly Lewis is a journalist whose work has appeared in the New York Times and the New York Times Magazine, Vanity Fair, Wired, Interview Magazine, Pitchfork, Elle, and Maclean’s, where she is a contributing editor. Her work has been recognized by the National Magazine Awards and the Digital Publishing Awards. She reports on city life, culture—including what people do online—politics, art and crime. She received the Dave Greber Freelance Writers Award for “The Murder of Ashley Wadsworth,” an investigative feature about a Canadian teenager who was killed by a man she met on social media, published by Maclean’s.
