The Case Manager Using ChatGPT for Progress Notes Is Not Your Biggest AI Risk

A care coordinator opens a new browser tab and types a client visit summary into ChatGPT to help draft a progress note. A home health RN pastes patient encounter observations into ChatGPT to draft a care plan update. A medical biller consults ChatGPT to advise on a patient’s claim denial. They’re not trying to break the rules. They’re new to this position and trying to meet documentation standards. They’re trying to get through a growing documentation backlog since their team is short-staffed. They’re trying to get questions answered without bothering their supervisors. No one told them what AI tools they can use. So they use the one they already know. This is what’s known as “shadow AI.”

When compliance teams discover this kind of usage, the instinct is to frame it as a training problem or a rogue employee. But that framing misses the structural issue. Staff reach for AI tools they’re familiar with outside of work because the organization has not yet provided approved alternatives, clear guidelines, or a pathway for them to suggest tools that could help. The governance is not in place yet.

The part that matters for leadership is that the care coordinator is not the organization’s biggest AI risk. The bigger risk is the gap between how fast AI tools become available vs how fast the oversight structures adapt to account for them. The buying cycle moves faster than the oversight cycle.

What the Data Shows About Shadow AI in Healthcare

In January 2026, Wolters Kluwer Health published a survey of more than 500 healthcare professionals and administrators. 40% reported encountering unauthorized AI tools in their organizations. 17% admitted to using them. The top reason? They needed a faster workflow and/or there was a lack of approved tools or insufficient functionality in the tools available to them.

These numbers align with what ECRI, the independent patient safety organization, flagged in its 2026 Top 10 Health Technology Hazards report. ECRI placed misuse of AI chatbots at the top of the list. Consumer chatbots like ChatGPT and Gemini, used by clinicians and staff without organizational oversight.

The cost is worth understanding. IBM’s 2025 Cost of a Data Breach report found that healthcare remains the most expensive industry for data breaches for the 14th consecutive year, averaging $7.42M per incident. When shadow AI is a contributing factor, it adds approximately $670K to the breach cost. 97% percent of organizations that experienced AI-related security incidents lacked proper access controls on AI systems. 63% lacked AI governance policies entirely.

None of these organizations set out to be negligent. This is what happens when there’s a structural gap between the tools staff have access to and the policies governing their use.

3 Things Missing When Shadow AI is Widespread

When we look at organizations where shadow AI is prevalent, we find the same structural gaps:

1.     There’s no approved tools list. Staff don’t know what they can use because the organization has never formally evaluated and approved AI tools for specific workflows. If no tools have been formally approved, there is nothing to train on and nothing to point to when a staff member asks what is allowed.

2.     There’s no prohibited tools list with clear rationale. Staff don’t know what they can’t use, or why. Staff don’t realize they’re violating an organizational policy without a document that names personal ChatGPT use specifically, explains that data entered may train the vendor’s models, and notes that no Business Associate Agreement is available.

3.     There’s no path for staff to suggest new tools. This is the gap that turns well-meaning employees into shadow AI users. When a care coordinator discovers a tool that saves 20 minutes per visit, she has 2 choices: use it without telling anyone, or do nothing. A staff-facing AI guide solves this by providing a pathway (e.g. talk to your supervisor about the tool and the workflow step it could improve, submit the tool for evaluation using the organization’s AI Decision-Making Checklist, etc.)

When you make it easy to do the right thing, staff do the right thing.

What a Practical Staff AI Guide Actually Looks Like

The document that closes this gap is not a 30-page policy manual. It is a 1-page staff reference that answers three questions: 1) What am I allowed to use?; 2) What am I not allowed to use?; 3) What do I do when something goes wrong?

If the organization has not yet formally approved any AI tools, the guide states that explicitly and explains that the list will be updated as tools complete evaluation. That clarity alone reduces shadow AI use because staff know the organization is working on it, not ignoring it.

The prohibited tools section is equally specific. It names the tools by name, like personal ChatGPT and Google Gemini on personal accounts, and unapproved browser extensions. It explains why each is prohibited. The reason is not "because we said so." The reason is that consumer AI tools may use entered data to train their models, no BAA is available, and the organization has no control over data retention. When staff understand the "why," compliance goes up.

The incident response section gives staff a pathway for when AI outputs are causing problems to the workflow (e.g. stop using the AI output, document what happened, report through the existing incident reporting system, etc.)

You Already Have Most of What You Need

If you run annual HIPAA compliance training, you already have 80% of the structure for AI staff training. The core messages overlap: do not enter sensitive data into unapproved tools, report incidents through the existing process, verify information before adding it to a patient record. Expanding that training to include a 30-minute module on your approved and prohibited AI tools list, the tool suggestion process, and the requirement to label AI-assisted content covers the remaining 20%.

If you maintain an incident reporting system, you already have the mechanism for AI incident reporting. Adding fields to the existing form captures the specific tool involved, whether the AI output was used before the error was identified, and whether this appears to be a recurring pattern.

Staff using their personal ChatGPT accounts is not the problem to solve. They are the signal that the governance structures have not caught up yet. And closing that gap does not require building something from scratch. It requires expanding what already exists by 1 document, 1 training module, and a handful of additional questions in the processes you already run.

Most organizations are closer to having this in place than they think.

How has your organization handled the discovery of unapproved AI tools? Tell us what you’ve found that works or doesn’t work.

 #HealthcareAI  #AIGovernance  #TrustworthyAI  #CommunityHealth  #AIReadiness