AI medical record risks: hallucination, omission, PHI and the BAA
Answer in brief: the 6 risks that matter are invented facts (hallucinations), dropped facts (omissions), flipped negatives, PHI sent to a service with no BAA, your data used to train someone's model, and bias rules for tools that steer patient care. The first 3 are caught by checking citations; the last 3 are settled in the contract before the first upload.
Limits: Vendor facts come from each vendor's own website, read 2026-11-12, and we haven't tested every tool; prices and features change, so check the vendor before you buy.
How often AI gets medical text wrong
The best published breakdown is Asgari and colleagues in npj Digital Medicine (2025). 50 doctors checked AI-written primary care notes sentence by sentence against the consultation transcripts: 450 notes, 12,999 note sentences, 49,590 transcript sentences (study). It measures note drafting from a transcript, not summarizing a 2,000-page file, so treat it as the best available signal, not your tool's error rate.
2 readings of the same numbers are both right. The error rate is low, and the authors note that human-written notes also average errors and omissions. But on a long file a low rate still produces errors you have to find, and omissions outnumber hallucinations by about 9 to 1 in this study. A tool that never invents anything can still leave out the 1 finding your opinion turns on.
The 6 risks and how to catch them
| Risk | What it looks like | How to catch it | When |
|---|---|---|---|
| Hallucination | A finding, date or drug that the source doesn't contain | Open the cited page for every line you rely on; distrust any line with no citation | Every file |
| Omission | A key result, prior injury or allergy left out | Ask targeted questions after reading the summary; scan the index of documents for sources the summary never cites | Every file |
| Flipped negative | "Denies numbness" becomes "numbness" | List the negatives your opinion depends on and check each against the source | Every file |
| PHI with no BAA | Records pasted into a consumer chatbot account | Signed BAA before the first upload, covering the vendor and its subprocessors | Once, before the trial |
| Training on your data | Your records improve the vendor's model | A contract term, ideally in the BAA, that customer data never trains a model | Once, at contract |
| Bias in care decisions | A risk score that uses race, sex, age or disability as inputs | Inventory decision support tools and their inputs, as 45 CFR 92.210 requires of covered entities | At purchase and yearly |
Why negatives and dates fail most
Negatives fail because language models predict likely text, and in a chart about back pain, "numbness" is a likely word whether or not the patient denied it. In the Asgari study, 30% of hallucinations were negations. Dates fail for a different reason: outside records carry several dates per page (fax header, print date, date of service, signature date), and the tool has to pick the right one. Both errors look plausible, which is what makes them dangerous. A made-up drug name catches the eye; "positive straight leg raise" where the source said negative doesn't.
Context errors are the third pattern: a fact that is true but attached to the wrong episode, the wrong side or the wrong patient in a file that contains a family member's records. Ask the vendor how the tool handles co-mingled records, then plant one in your test file.
PHI, the BAA and model training
Under HIPAA, a vendor that creates, receives, maintains or transmits electronic PHI for you is a business associate. HHS says a cloud service is a business associate even when it stores only encrypted data and has no key, and that the covered entity must sign a BAA with it (HHS guidance). Before any upload, get written answers to 6 questions:
- Will you sign a BAA before the trial, and does it cover your subprocessors, including the AI model provider?
- Does any customer data, including de-identified data, train or tune a model? Where is that written?
- Where is the data processed and stored, and can it leave the US?
- How long do you keep uploaded files and outputs after a case is closed or deleted?
- Can you share a current SOC 2 report or summary?
- Who on your staff can see our files, and is access logged?
A vendor that answers "it's HIPAA compliant" without a signed BAA hasn't answered. Patient-facing apps sit outside this: when patients connect their own records to ChatGPT Health or to Claude through athenahealth's connector, they are directing their own data, and OpenAI says connected records are not used to train its foundation models (OpenAI help center).
Bias rules for tools that steer care
Section 1557 of the Affordable Care Act now reaches software. Under 45 CFR 92.210, a covered entity must not discriminate through patient care decision support tools, must make reasonable efforts to identify tools whose inputs measure race, color, national origin, sex, age or disability, and must make reasonable efforts to mitigate the risk (45 CFR 92.210). A record summary rarely counts as such a tool. A model that scores readmission risk or prioritizes referrals from the record might. Hospital teams should put AI record tools on the same inventory as other decision support.
A 10-minute check routine for any AI output
- Find the 5 facts your opinion or plan depends on. Open the cited page for each.
- Check every negative among them against the source wording.
- Check the date of service on each cited page, not the fax or print date.
- Ask the tool 2 questions it should answer "no" to, phrased 2 ways each.
- Scan the document index for sources the summary never cites. Ask about each.
- Note in your file which output you relied on and what you checked. If you sign, the opinion is yours.
For a full trial of a new tool, use the scoring sheet, which scores these same risks on a file you already know.